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Preface 


PREFACE 


This  document  is  one  of  a  family  of  education¬ 
al  guides  written  from  a  Department  of 
Defense  (DOD)  Perspective  (i.e.,  non-ser¬ 
vice  peculiar).  These  books  are  intended 
primarily  for  use  in  the  courses  at  the  Defense 
Systems  Management  College  (DSMC)  and 
secondarily  as  a  desk  reference  for  program 
and  project  management  personnel.  The 
books  arc  written  for  current  and  potential 
DOD  Acquisition  Managers  who  have  some 
familiarity  with  the  basic  terms  and  defini¬ 
tions  of  the  acquisition  process.  It  is  intended 
to  assist  both  the  Government  and  industry 
personnel  in  executing  their  management 
responsibilities  relative  to  the  acquisition  and 
support  of  defense  systems.  This  family  of 
technical  guidebooks  includes: 

Integrated  Logistics  Support  Guide',  F’irst  Edi¬ 
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Systems  Engineering  Management  Guide', 
Second  Edition:  Dec  1986 

Department  of  Defense  Manufacturing 
Management  Handbook  for  Program 
Managers',  Second  Edition:  July  1984 
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COPY  AND  MICROFICHE,  AUTH:  DSMC  PRESS 
(MR  BALL  703-805-2852)  PER  TELECON, 

21  MAY  93-  CB 


Test  and  Evaluation  Management  Guide, 
March  1988 

Acquisition  Strategy  Guide,  First  Edition,  July 
1984 

Subcontracting  Management  Handbook,  First 
Edition,  1988 
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CHAPTER  1 


INTRODUCTION 


Mission  Critical  Computer  Resources 
(MCCR)  refers  to  the  totality  of  computer 
hardware  and  computer  software  that  is  in¬ 
tegral  to  a  weapon  system  along  with  the  as¬ 
sociated  personnel,  documentation,  supplies 
and  services.  A  natural  question  to  ask  is  "Why 
should  a  program  manager  be  that  interested 
in  MCCR."  The  answer  is  fourfold.  First, 
software  for  weapon  systems  is  on  the  "critical 
path"  of  system  development.  If  software 
development  falls  behind  schedule,  the  entire 
weapon  system  development  will  also  fall  be¬ 
hind  schedule.  Second,  software  can  produce 
development  problems  of  sufficient  mag¬ 
nitude  to  result  in  costly  program  overruns.  It 
is  not  uncommon  for  software  development 
costs  to  exceed  initial  budget  estimates  by  as 
much  as  50%  to  100%.  Third,  the  perfor¬ 
mance  of  modern  weapon  systems  is  largely 
dependent  on  the  quality  of  their  computer 
resources;  the  system  is  only  as  good  as  its 
software.  Fourth,  it  is  an  established  fact  that, 
if  the  software  development  falls  behinu 
schedule,  the  development  lead  times  cannot 
be  shortened  simply  by  applying  more  resour¬ 


ces.  Money  can’t  fix  the  problem  —  only  time 
can  [1].  Once  a  program  falls  behind,  little 
can  be  done  to  save  it! 

Management  of  MCCR  development  cannot 
be  ignored  or  delegated.  If  the  program 
manager  leaves  all  MCCR  management  con¬ 
siderations  to  the  development  contractor, 
there  is  the  strong  possibility  that  the  software 
development  will  encounter  significant  dif¬ 
ficulties.  The  management  of  MCCR 
development  may  be  compared  with  piloting 
an  aircraft.  Without  the  proper  application  of 
the  necessary  control,  it  is  extremely  unlikely 
that  the  aircraft  will  safely  reach  the  intended 
destination.  Without  the  appropriate 
management  directives,  it  is  unlikely  that 
MCCR  development  will  result  in  a  suitable 
product.  There  are  no  autopilots  for  MCCR 
development.  Effective  MCCR  acquisition 
management,  like  piloting,  is  a  difficult  task 
but,  with  proper  knowledge  and  care,  not  im¬ 
possible  [2].  This  guide  will  only  cover  the 
basics  by  providing  enough  background  to 
enable  straight  and  level  flight.  Aerobatics 
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(i.e.,  unconventional  developments)  depend 
on  greater  mastery  of  the  fundamentals, 
which  is  outside  the  scope  of  this  guidebook. 

The  actual  MCCR  development  will  be  ac¬ 
complished  by  a  system  development  contrac¬ 
tor.  On  occasion,  the  "contractor"  may  be 
another  DOD  agency.  The  development  con¬ 
tractor  has  the  responsibility  of  delivering  a 
software  product  that  meets  all  contractual 
requirements.  Unfortunately,  it  is  not  pos¬ 
sible  to  specify  precisely  and  completely  in  a 
contract  all  the  characteristics  of  the  final 
software  product  and  its  development 
process.  Experience  has  shown  that  the  dif¬ 
ference  between  successful  and  unsuccess¬ 
ful  development  efforts  is  often  the  rigor  and 
timeliness  of  the  direction  given  to  the  con¬ 


tractor  by  the  procuring  agency’s  program 
management  organization  [2]. 

REFERENCES 

1.  Meinke,  George  W.,  Airborne  Software  Ac¬ 
quisition  Management... A  Guide  for  New 
Software  Managers,  Air  Command  and  Staff 
College  Report  Number  82-1685,  Air  Univer- 
sitv,  Maxwell  AFB,  A1  36112. 

2.  Rubey,  Raymond  }.,A  Guide  to  the  Manage¬ 
ment  of  Software  in  Weapon  Systems,  Prepared 
by  Softech  for  the  U.S.  Air  Force  Aeronauti¬ 
cal  Systems  Division  and  the  U.S,  Army  Avia¬ 
tion  Systems  Command,  2nd  Edition,  March 
1985. 
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INTRODUCTION  TO  COMPUTER 
RESOURCES 


2.1  HISTORICAL  PERSPECTIVE 

The  development  of  computer  software  as  a 
recognized  activity  is  less  than  40  years  old. 
During  the  infancy  of  digital  computers 
(1950s)  all  software  or  computer  programs 
were  dev  'oped  by  engineers  or  scientists  as 
an  adjunct  to  their  work  with  computers.  In 
the  early  1960s,  computer  technology  was 
usually  taught  under  the  auspices  of  university 
or  college  electrical  engineering  depart¬ 
ments.  It  wasn’t  until  the  late  1960s  that  com¬ 
puter  science  departments  were  being 
established  as  separate  entities.  The  term 
"software  engineering"  was  not  coined  until 
1968  when  the  term  was  used  as  a  theme  of 
several  workshops  held  in  West  Germany  and 
Italy  to  address  the  growing  problems  as¬ 
sociated  with  software  development  [1].  So 
unlike  other  disciplines,  such  as  electrical  en¬ 
gineering,  software  engineering  is  a  relative 
newcomer.  Because  of  this,  its  practitioners 
do  not  have  at  their  disposal  the  wealth  of 
time-tested  practices,  procedures  and  tools  so 
readily  available  to  its  sister  disciplines.  In 


circuit  design,  for  example,  an  electrical 
engineer  can  use  off-the-shelf  components 
and  modules  with  the  necessary  charac¬ 
teristics  to  build  large  portions  of  a  system. 
Off-the-shelf  components  or  modules  are  not 
widely  available  to  the  software  engineer. 
Quite  often  attempts  to  use  existing  software 
lead  to  major  problems  if  the  designers  are 
not  careful  when  integrating  the  existing 
software  into  their  design.  Designers  must 
fully  understand  all  of  the  characteristics  of 
existing  software  as  well  as  its  overall  perfor¬ 
mance  and  reliability.  Software  engineering 
is  still  in  its  infancy.  New  innovations  are 
being  introduced  every  day,  but  they  are  more 
evolutionary  in  nature  than  revolutionary. 
Many  of  the  problems  associated  with 
software  engineering  are  due  to  the  relative 
immaturity  of  the  discipline.  There  is  still  too 
much  art  and  craft  and  not  enough  engineer¬ 
ing  in  software  development;  although  the 
trend  is  definitely  changing. 

The  introduction  and  growth  of  digital  sys¬ 
tems  in  the  DOD  parallel  the  introduction 
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and  growth  of  digital  systems  in  the  commer¬ 
cial  market  place.  In  fact,  the  stringent  re¬ 
quirements  of  military  systems  often 
spearheaded  the  development  of  computers 
and  software  throughout  the  industry,  espe¬ 
cially  during  the  early  years  of  computer 
development.  In  the  early  195()s  all  weapon 
systems  were  analog  and  it  wasn’t  until  the 
mid-fifties  that  digital  systems  were  intro¬ 
duced  into  weapon  systems.  During  the  six¬ 
ties  there  was  a  rapid  incorporation  of  digital 
systems  with  an  almost  exponential  growth 
occurring  in  the  seventies.  Some  of  the 
reasons  for  this  rapid  growth  were: 

(a)  Advances  in  integrated  circuits  (ICs),  the 
basic  building  blocks  of  electronic  equipment 
and  digital  computers.  ICs  were  being 
developed  with  ever  increasing  capabilities 
and  an  accompanying  decrease  in  power  re¬ 
quirements,  size  and  cost. 

(b)  The  introduction  of  the  microprocessor, 
which  is  essentially  a  computer  on  a  chip, 
allowed  designers  to  replace  many  pieces  of 
hardware  with  a  single  component  roughly  the 
size  of  a  postage  stamp. 

(c)  The  ever  increasing  Soviet  threat  and  the 
need  to  counter  it  in  ihe  face  of  decreasing 
defense  budgets.  This  drove  the  services  to 
build  fewer  but  smarter  and  technically  supe¬ 
rior  weapons  relying  more  and  more  on  com¬ 
puters  and  software. 

(d)  The  realization  that  software  is  inherently 
more  flexible  than  hardware  and  better  able 
to  accommodate  the  ever  changing  threat. 

(e)  The  tremendous  advances  made  in  the 
commercial  marketplace  in  computers  and 
software. 

Today,  all  weapon  systems  are  dependent  on 
computers  and  software.  This  phenomenal 


growth  of  digital  systems  in  aircraft,  for  ex¬ 
ample,  is  shown  in  Figure  2-1.  In  1966  the 
FB-111  required  an  on-board  computer 
memory  of  roughly  60,000  words  but  by  1988 
the  B-IB  Bomber  was  approaching  on-board 
computer  memory  requirements  of  about  2.5 
million  words.  Current  and  future  systems 
will  greatly  exceed  these  memory  require¬ 
ments  with  large  scale  software  systems  being 
the  norm. 

2.2  SOFTWARE  IMPACT  ON  SYSTEM 

What  does  the  curve  in  Figure  2-1  tell  us?  If 
one  were  to  use  the  analogy  of  constructing  a 
brick  wall,  one  can  say  that  a  3000  square  foot 
wall  requires  about  three  times  as  many  bricks 
as  a  1000  square  foot  wall.  It  is  not  necessarily 
more  difficult  to  build,  it  just  takes  longer. 
Unfortunately  this  analogy  breaks  down  when 
it  comes  to  software. 

The  impact  of  software  on  system  design  and 
development  is  illustrated  in  Figure  2-2  [2]. 
The  dashed  lines  represent  the  average  in¬ 
fluence  of  either  software  or  hardware  on 
system  design  and  development  while  the 
solid  lines  on  either  side  represent  the  maxi¬ 


mum  and  minimum  range  of  intluence.  It  is 
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Fig.  2-1  Growth  of  Aircraft  Systems 
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Fig.  2-2  Software  Impact  on  System 
Design/Development 

very  clear  that  in  1950  software  had  no  in¬ 
fluence  on  weapon  systems  design.  This  is 
because  these  systems  contained  no  digital 
hardware.  By  1980,  however,  the  relative  in¬ 
fluence  of  software  on  system  design  averaged 
about  50%  with  some  systems  being  in¬ 
fluenced  by  as  much  as  70%  or  as  little  as  30%. 
This  means  that  software  considerations  af¬ 
fected  overall  system  design  and  development 
about  50%  of  the  time.  System  engineers 
could  no  longer  make  hardware  design 
decisions  without  considering  the  software 
implications.  As  can  be  seen,  the  trend  seems 
to  be  for  an  increasing  role  tor  software. 
What  the  figure  shows  is  that  software  is  no 
longer  merely  a  part  of  the  system;  software 
has  become  a  system  in  its  own  right  and  has 
assumed  the  integration  function  for  the 
various  subsystems  of  a  weapon. 

2.3  LIFE  CYCLE  COST  TRENDS 

Figure  2-2  implies  that  hardware  has  been 
traded  for  software.  Why  should  that  be  a 
problem?  The  problem  is  one  of  cost  as 
shown  in  Figure  2-3  [3].  This  figure  shows  that 
over  the  last  30  years  the  cost  of  the  software, 
as  a  percentage  of  total  computer  resources 
cost,  has  been  growing  by  leaps  and  bounds 
whereas  the  associated  computer  hardware 


has  been  decreasing  by  an  equally  dramatic 
percentage.  A  personal  computer  (PC)  of 
today  has  more  power  than  the  large  com¬ 
puters  which  were  the  workhorses  of  early 
space  programs  and  it  costs  a  mere  fraction  of 
its  predecessor.  In  addition,  a  PC  will  sit  on  a 
corner  of  your  desk  while  the  early  computers 
occupied  large  rooms  and  required  thousands 
of  watts  of  electricity  and  tons  of  air  condition¬ 
ing. 


A  word  of  caution  is  necessary.  Figure  2-3 
only  refers  to  large,  complex,  military  systems 
which  are  produced  in  limited  numbers.  In 
commercial  products  software  is  generally 
simpler  and  items  are  manufactured  in  very 
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Fig,  2-3  Life  Cycle  Cost  Trends 

large  numbers.  Therefore  the  cost  of  software 
used  in  appliances,  automobiles,  toys,  and 
other  commercial  products  actually  comprise 
but  a  small  fraction  of  the  total  cost  of  these 
items  [4].  This  is  depicted  in  Figure  2-4. 


The  decreasing  size  of  computer  hardware, 
along  with  their  increasing  capabilities,  ha.s 
resulted  in  an  explosion  of  applications  in  our 
weapon  systems.  Software  not  only  performs 
many  of  the  functions  previously  performed 
by  specialized  hardware,  it  also  performs 
many  of  the  functions  which  would  be  impos¬ 
sible  or  impractical  to  perform  with  just  dedi¬ 
cated  hardware.  This  tremendous  use  of 
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computers  has  come  at  a  price.  As  Figure  2-3 
shows,  the  ratio  of  computer  hardware  and 
software  expenditures  has  changed  from  a 
ratio  of  80%  hardware  and  20%  software  in 
1960  to  a  ratio  of  20%  hardware  and  80% 
software  in  1980.  There  has  been  an  equally 
dramatic  increase  in  the  costs  associated  with 
supporting  the  software  once  the  system  is 
delivered.  The  primary  reason  software 
development  and  support  are  so  expensive  is 


that  both  functions  are  extremely  labor  inten¬ 
sive.  It  is  an  ironic  fact  that  an  industry  which 
has  provided  the  means  for  other  segments  of 
industry  to  automate,  has  itself  failed  to 
automate.  There  are  very  few  machines  and 
computer  software  that  will  automatically 
generate  computer  programs  directly  from  a 
set  of  requirements.  Those  that  exist  are 
limited  to  very  special  applications. 

The  cost  of  DOD  software  is  immense  as  can 
be  seen  from  the  chart  in  Figure  2-5,  Accord¬ 
ing  to  this  Electronics  Industries  Association 
study  [5],  by  the  year  1990  the  cost  of  software 
alone  will  be  approximately  $25.6  billion.  To 
put  that  figure  in  perspective,  the  total  price 
tag  for  the  B-IB  Bomber  fleet  was  around  $20 
billion  in  1981  dollars.  That  included  100 
aircraft,  the  initial  spares,  the  weapon  system 
crew  and  maintenance  trainers  and  the  initial 
logistics  support.  The  DOD  could  greatly  in¬ 
crease  the  strategic  bomber  fleet,  for  the 
amount  of  money  being  spent  on  software 
today. 


Fig.  2-5  DoD  Embedded  Computer  Market 

In  contrast,  the  amount  of  money  being  spent 
on  computer  hardware  has  barely  increased. 
When  one  considers  that  computer  hardware 
is  more  powerful  today  then  ever  before,  the 
cost  for  comparable  performance  has  actually 
decreased  immensely.  Many  factors  have 
contributed  to  the  decreasing  cost  of  com¬ 
puter  hardware  but  certainly  automation  has 
been  a  major  contributor.  Unfortunately  it 
still  requires  a  person  to  program  a  computer 
and  programmers  and  other  software 
specialists  are  expensive. 

2.4  EVOLUTION  OF  DIGITAL  SYSTEMS 


If  one  were  to  make  a  comparison  of  Korean 
War  vintage  aircraft  with  modern  day  aircraft. 


Fig.  2-6  Evolution  of  Fighter  Aircraft 
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one  realizes  that  both  aircraft  contain  the 
same  types  of  functional  systems  (Figure  2-6). 
A  Korean  War  vintage  aircraft  contained 
flight  controls,  navigational  gear,  radar,  etc. 
A  modern  day  aircraft  has  the  same  type  of 
systems  except  it  now  has  a  flight  control  sys¬ 
tem,  an  inertial  navigation  system  or  INS,  a 
radar,  etc.  Let’s  examine  some  of  these  sys¬ 
tems  in  greater  detail. 

A  1952  F-86  Sabre,  for  example,  had  a  stick 
which  was  physically  attached  to  mechanical 
linkages.  These  in  turn  were  attached  to  the 
various  hydraulic  actuators  and  control  sur¬ 
faces  such  as  elevators.  When  a  pilot  moved 
the  stick,  there  would  be  an  accompanying 
movement  of  the  control  surfaces  because  of 
the  mechanical  connection.  The  amount  of 
force  required  to  move  the  stick  depended  on 
how  fast  the  aircraft  was  flying  and  its  attitude. 
Contrast  this  with  the  latest  version  of  the 
F-16  where  the  mechanical  linkages  have 
been  replaced  with  electrical  wires  and 
motors.  Movement  of  the  stick  creates  an 
electrical  signal  which  travels  down  the  wires 
and  activates  the  motors  to  physically  move  a 
control  surface.  There  may  also  be  two  or 
m.ore  wires  to  provide  redundancy  and,  since 
there  is  no  feedback  from  mechanical 
linkages,  a  means  to  artificially  give  the  pilot 
"a  feel”  for  moving  the  control  surfaces. 

All  of  this  is  done  with  modern  computers. 
The  signals  traveling  down  the  wire  are  digital 
in  nature  and  the  redundancy  checks  and  the 
artificial  "feel"  are  all  controlled  by  digital 
computers.  Furthermore,  the  computer  gives 
the  flight  control  system  the  ability  to  "fly"  the 
aircraft  in  ways  not  possible  if  a  pilot  were  the 
controlling  element.  The  X-29,  the  forward 
swept-wing  experimental  aircraft,  would  be 
virtually  impossible  for  a  pilot  to  fly  without 
the  complex  computer  dependent  flight  con¬ 
trol  system.  The  movement  of  the  stick  allows 
the  pilot  to  indicate  where  he  wants  the 


aircraft  to  go,  but  the  computer  actually  flies 
the  airplane. 

An  F-86,  for  example,  carried  navigational 
gear  on  board  to  allow  the  pilot  to  find  his  way 
to  his  destination.  This  consisted  of  a  mag¬ 
netic  compass,  an  altimeter  for  altitude  in¬ 
dication  and  perhaps  some  kind  of  radio 
direction  finding  equipment.  With  this 
navigational  gear  the  pilot  was  able  to 
navigate  using  dead  reckoning  techniques. 
Today  we  have  an  Inertial  Navigation  System 
(INS)  comprised  of  gyroscopes,  ac¬ 
celerometers,  and  computers  to  perform  the 
same  function.  The  INS  is  aligned  before 
takeoff  and  it  allows  the  pilot  to  accurately 
navigate  from  one  point  to  another.  This  is 
only  possible  because  of  the  computer.  Using 
a  known  model  of  the  error  sources  within  the 
INS,  the  computer  uses  a  mathematical  tech¬ 
nique  known  as  Kalman  Filtering  to  keep 
track  of  the  aircraft’s  exact  position  over  time. 

The  Korean  War  vintage  cockpit,  packed  full 
of  instruments,  has  been  replaced  with  a  cock¬ 
pit  containing  just  a  few  instruments  and  con¬ 
trols  and  display  screens.  Any  information 
required  by  the  pilot  is  simply  displayed  on  the 
screens  at  the  push  of  a  button,  anything  from 
attitude  indications  to  the  status  of  weapons. 
They  are  all  under  the  control  of  computers. 
Furthermore,  there  are  now  new  subsystems 
that  would  not  be  possible  without  digital  sys¬ 
tems:  diagnostic  systems  that  can  display  the 
health  of  all  the  major  subsystems  and  "expert 
systems"  that  provide  the  pilot  with  informa¬ 
tion  on  the  various  options  available  during  a 
particular  mission. 

In  summary,  one  can  say  that  computers  and 
software  have  introduced  a  whole  new  dimen¬ 
sion  to  our  weapon  systems;  improved  system 
performance:  become  an  aid  to  the  decision 
making  process;  expanded  the  capabilities  of 
the  human  operator  and  in  many  cases 
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replaced  the  human  operator.  In  short,  they 
have  dramatically  enlarged  the  performance 
envelope  of  weapon  systems  far  beyond  what 
was  possible  less  than  30  years  ago. 

Without  modern  computers  and  the  as¬ 
sociated  software,  modern  weapon  systems 
would  not  exist.  Weapon  systems  have 
evolved  from  systems  where  computers 
played  a  very  minor  role  to  systems  where 
their  very  existence  depends  entirely  on  com¬ 
puters. 

2.5  WEAPON  SYSTEM  SOFTWARE 


The  di.scussions  so  far  have  centered  on  the 
computers  and  the  software  that  are  em¬ 
bedded  in  a  weapon  system  and  are  an  integral 
part  of  that  system.  There  is,  however,  a 
whole  host  of  software  associated  with  every 
weapon  system  (Figure  2-7)  that  is  not  em¬ 
bedded  in  the  system  but  is,  nevertheless,  ab¬ 
solutely  essential. 


The  software  that  has  been  discussed  so  far  is 
referred  to  as  flight  software  but  a  wealth  of 
other  software  is  required  to  support  a 
weapon  system.  During  ground  and  flight 
testing,  extensive  data  reduction  computers 
and  software  are  required  to  aid  in  the  analysis 


of  literally  millions  of  data  points  generated 
in  a  major  test  program. 

In  ( .'der  to  train  the  various  maintenance 
crews,  various  subsystem  trainers  must  be 
developed  many  of  which  require  literally 
hundreds  of  thousands  of  lines  of  code  and 
numerous  computers,  both  large  and  small. 
Two  examples  are  avionics  maintenance 
trainers  and  weapons  load  trainers.  Opera¬ 
tional  analysis  personnel  require  very  large 
scenario  software  to  perform  tactics  and  war 
planning.  Extensive  and  complex  battle 
management  software  may  be  required  to 
develop  mission  planning.  Operational  crews 
usually  train  on  large  software  intensive 
weapon  system  trainers  whose  functions  are 
dependent  on  millions  of  lines  of  software 
instructions.  For  logistics  support  of 
electronic  equipment,  automatic  test  equip¬ 
ment  (ATE)  and  its  associated  software  must 
be  developed  along  with  hundreds  of  software 
packages  called  Test  Program  Sets  (TPSs). 
TPSs  allow  technicians  to  isolate  and  repair 
failed  electronic  components.  Lastly,  mis¬ 
sion  preparation  software  may  also  be  re¬ 
quired  for  use  by  operational  crews  in 
planning  and  carrying  out  their  missions. 

2.6  CURRENT  STATE  OF  AFFAIRS 

Examination  of  the  current  state  of  affairs 
with  military  weapon  systems  reveals  some 
very  unpleasant  facts: 

(a)  Most  new  systems  are  extremely  complex. 
This  is  due  to  a  combination  of  several  factors: 

-  extremely  demanding  requirements,  which 
tend  to  drive  designers  towards  complex  solu¬ 
tions; 

-  tight  schedules  and  even  tighter  budgets, 
which  tend  to  negate  elegant  and  simpler 
solutions; 
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-  and,  unfortunately,  too  many  contractors 
not  fully  skilled  in  software  engineering  tech¬ 
niques  tend  to  populate  the  "lowest  bidder" 
category.  These  contractors  seem  to  thrive 
under  our  current  procurement  laws  and 
regulations. 

(b)  Digital  systems  are  now  the  heart  and  soul 
of  all  new  weapon  systems.  The  flexibility 
offered  by  digital  systems  cannot  be  remotely 
approached  by  analog  systems.  This  trend  will 
continue  for  the  foreseeable  future. 

(c)  Most  systems  are  delivered  late,  have  cost 
overruns,  rarely  meet  performance  require¬ 
ments  upon  initial  delivery  and  are  often 
ridiculously  expensive  to  maintain. 

It  would  be  very  unfair  to  blame  all  of  these 
unpleasant  facts  just  on  digital  systems  and 
software,  but  it  is  generally  recognized  that 
software  is  a  major  contributor,  and  often  the 
only  contributor,  to  these  problems.  Software 
has  become  the  Achilles  heel  of  weapon  sys¬ 
tems.  Not  only  is  it  in  the  critical  path  of  the 
system  development  process  but  system  per¬ 
formance  is  dependent  on  the  system 
software. 

2.7  HISTORICAL  CONTRIBUTORS 

One  of  the  major  contributors  to  the  problems 
associated  with  software  development  has 
been  loose  and  very  often  nonexistent 
management  oversight.  Since  most  program 
managers  know  little  or  nothing  about 
software,  they  concentrate  their  efforts  on 
hardware  or  system  issues  and  often  leave 
software  management  to  managers  who  are 
not  always  part  of  the  mainstream  decision 
making  process.  They  get  involved  only  when 
software  starts  affecting  the  overall  schedule 
and  by  then  it  is  usually  too  late.  This  is 
changing  because  a  policy  of  "benign  neglect" 
is  no  longer  acceptable. 


Another  problem  is  that  some  software 
managers  lack  relevant  experience.  This  is  a 
universal  problem  with  no  quick  solutions  in 
sight.  Experienced  software  managers  within 
the  government  are  a  scarce  commodity.  In¬ 
dustry  seems  to  lure  the  good  ones  away  and 
those  that  remain  and  rise  to  management 
positions  are  not  necessarily  the  most  ex¬ 
perienced. 

A  major  problem  that  has  plagued  the  DOD 
in  the  past  has  been  the  uneven  application  of 
standards  and,  in  some  cases,  the  lack  of 
standards.  As  for  the  former,  too  often  a  con¬ 
tract  simply  calls  out  all  the  applicable  stand¬ 
ards  without  regard  to  the  fact  that  many  of 
them  are  contradictory  or  even  unnecessary. 
All  standards  and  regulations  should  be 
tailored  for  each  program.  It  wasn’t  until 
recently,  that  common  programming  stand¬ 
ards  were  mandated  for  all  the  services. 

Throughout  the  1960s  and  1970s  the  total 
number  of  programming  languages  used  for 
military  systems  numbered  in  the  hundreds 
and  none  of  them  were  compatible  with  each 
other.  Most  systems  used  their  own  languages 
and  their  own  computers  so  that  transpor¬ 
tability  across  systems  was  nonexistent.  To  a 
great  extent  many  systems  still  suffer  from  this 
problem. 

Another  contributor  to  the  software  problem 
has  been  the  almost  endemic  lack  of  a  dis¬ 
ciplined  engineering  approach  to  software 
development.  The  better  developers  have  the 
necessary  discipline  to  do  a  good  job  but  even 
they  encounter  problems.  Unfortunately, 
many  software  developers  only  pay  lip  service 
to  using  modern  software  development 
methodologies  such  as  top-down  structured 
design,  object-oriented  design  (OOD),  in¬ 
cremental  development,  software  metrics, 
stringent  configuration  management  prac¬ 
tices,  and  integrated  software  engineering  en- 
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vironments.  They  may  write  convincing 
proposals  but  they  can’t  always  deliver.  On 
the  government  side,  the  problem  is  com¬ 
pounded  when  the  contractor  is  not  forced  to 
follow  a  disciplined  engineering  approach. 
Too  often,  engineering  discipline  is  sacrificed 
to  those  holiest  of  sacred  cows— schedule  and 
cost. 

Somewhat  related  is  the  fact  that  competent 
software  developers  are  not  sufficient  in  num¬ 
bers  to  satisfy  the  demands  of  both  the 
military  and  the  civilian  market  place.  There 
are  more  software  projects  than  there  are 
competent  software  developers,  so  the  less 
skillful  fill  the  vacuum. 

Lastly  one  must  remember  that  the  software 
that  is  developed  for  weapon  systems  is  the 
most  difficult  and  most  challenging  type  of 
software.  Some  of  the  reasons  for  this  dif¬ 
ficulty  are  the  following: 

(a)  Most  weapon  systems  have  real-time  re¬ 
quirements  which  add  an  additional  level  of 
complexity.  The  software  has  to  respond  al¬ 
most  instantaneously  and  correctly,  in  spite  of 
noise  and  other  types  of  interference  that  can 
seriously  degrade  a  system. 

(b)  Most  weapon  systems  have  a  requirernt.nt 
for  fault-free  operation  or  some  level  of  fault 
tolerance.  This  adds  additional  overhead  to 
the  software  since  more  checks  and  redun¬ 
dant  capabilities  have  to  be  added.  These 
requirements  run  counter  to  the  require¬ 
ments  for  speed,  simplicity,  and  real-time 
response. 

(c)  Because  of  the  complexity,  many  software 
developments  stretch  over  periods  of  three  to 
five  years.  During  this  time  there  is  usually 
significant  personnel  turnover,  especially  in 
the  government.  This  results  in  loss  of  con¬ 
tinuity  and  provides  ample  opportunity  for 


new  "shakers  and  movers"  to  express  their 
leadership  and  managerial  "styles".  It  also 
provides  plenty  of  time  for  Congress  to  cut, 
slice,  batter  and  reap  havoc  with  the  budget. 
Although  this  is  also  true  for  hardware,  it  has 
a  more  severe  impact  on  software  develop¬ 
ment. 

(d)  Requirements  are  usually  not  finalized 
until  late  in  the  development  cycle.  This  is 
partly  due  to  the  evolving  threat  and  partly 
due  to  the  long  development  period.  The  end 
result  is  that  designers  are  shooting  at  a 
moving  target. 

(e)  No  human  endeavor  is  entirely  free  of 
politics.  The  program  office  is  no  exception, 
A  program  manager  must  not  only  deal  with 
management  and  technical  problems  but  also 
learn  to  navigate  the  more  dangerous  waters 
of  internal  and  external  politics, 

(f)  Compounding  the  problem  even  further  is 
the  sad  reality  that  computer  re.sources  tech¬ 
nology  is  rapidly  changing.  Technology  that  is 
state-of-the-art  at  the  beginning  of  a  major 
weapon  system  development  is  often  obsolete 
by  the  time  the  system  is  fielded.  A  program 
manager  must  be  able  to  properly  balance  the 
risk  as.sociated  with  using  technology  that  is  at 
the  cutting  edge,  but  which  is  not  yet  fully 
proven,  with  the  risks  associated  with  using 
more  proven  but  less  capable  technology. 

2.8  MANAGEMENT  GUIDANCE 

Program  managers,  as  well  as  software  per¬ 
sonnel,  need  to  be  educated  and  trained.  The 
Program  Managers  Course  at  the  Defense 
Systems  Management  College  is  an  example 
of  this  type  of  training.  This  education  and 
training  must  be  made  available  to  all  pro¬ 
gram  office  personnel.  Even  if  one  is  not 
directly  involved  with  software,  that  person 
should  still  have  an  appreciation  for  the  dif- 
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ficulties  involved  since  all  future  hardware 
development  will  be  impacted  by  software. 
Program  managers  must  allow  their  software 
personnel  to  attend  courses  and  seminars  so 
that  they  can  better  learn  the  process.  This 
training  is  especially  valuable  for  junior  and 
middle  level  software  managers  and  en¬ 
gineers,  many  of  who  have  little  or  no  formal 
training  in  software  engineering.  Program 
managers  must  never  use  the  excuse  that  "we 
are  too  busy  to  let  them  go  now."  They  will 
always  be  too  busy.  Make  the  time  and  let 
program  personnel  attend  training  classes. 

All  program  personnel  must  lose  their  fear  of 
software.  There  is  nothing  magical  about 
computers  and  software  as  long  as  time  is 
taken  to  learn  at  least  some  of  the  basics. 
Software  and  computer  illiteracy  can  no 
longer  be  tolerated  in  a  program  office. 

Software  and  hardware  standards  should  be 
intelligently  applied.  They  should  be 
scrutinized  and  tailored  to  a  specific  program. 
Most  standards  are  written  to  cover  the  entire 
waterfront  and  particular  programs  only  deal 
with  a  portion  of  that  waterfront.  Failure  to 
do  so  will  create  confusion  and  will  eventually 
impact  those  sacred  cows-schedule  and  costs. 

F"rom  day  one,  program  managers  must  pay 
attention  to  software  and  ensure  that  program 
personnel  are  doing  the  job  of  enforcing  the 
developer  to  follow  a  disciplined  process.  The 
chief  software  person  must  be  made  visible  by 
having  to  report  to  the  program  manager  on  a 
weekly  basis  and  on  a  daily  basis  during  criti¬ 
cal  periods.  This  person  should  be  totally 
aware  of  all  of  the  developer’s  major  activities 
and  have  the  facts  readily  available. 

The  program  manager  must  make  absolutely 
sure  that  program  personnel  actually  read  and 
critically  evaluate  all  software  documents 


submitted  by  the  developer.  If  they  are  rub¬ 
ber-stamping  documents,  they  should  be 
replaced.  Reviews  such  as  Preliminary  Design 
Reviews  (PDRs)  and  Critical  Design  Reviews 
(CDRs)  should  be  delayed  until  the  proper 
documents  have  been  thoroughly  reviewed. 
A  developer  should  not  be  allowed  to  slip 
through  a  gate  until  it  has  satisfied  all  the 
requirements  for  going  through  that  gate;  or 
until  everyone  is  fully  aware  of  the  risks  in¬ 
volved  by  proceeding. 

As  will  be  seen  in  a  later  section  of  this  guide, 
early  and  thorough  planning  is  the  only  way 
that  the  software  problems  can  be  minimized. 
This  planning  starts  during  the  concept  ex¬ 
ploration/definition  phase  and  continues  to 
some  degree  until  the  system  is  no  longer 
used. 

There  are  no  magical  solutions!  Good 
software  development  requires  extensive 
planning  and  thorough  vigilance.  There  are 
no  short  cuts  or  cookbook  solutions! 
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CHAPTERS 


TECHNICAL  FOUNDATIONS 


3.1  INTRODUCTION 

This  chapter  addresses  the  basics  of  computer 
hardware  and  software  by  describing  how  a 
computer  works  and  by  defining  the  concepts 
of  computer  programs  and  languages.  These 
basics  are  intended  to  provide  the  uninitiated 
with  an  understanding  and  appreciation  for 
why  a  software  development  project  must  fol¬ 
low  a  logical  and  proven  process. 

This  chapter  also  provides  a  brief  technical 
description  of  the  Ada  Programming  Lan¬ 
guage  and  a  brief  explanation  of  its  technical 
and  management  benefits. 

3.2  INSIDE  THE  COMPUTER 

In  general  terms,  a  computer  is  a  device  which 
receives  or  "senses"  data  through  input 
devices,  processes  that  data  and  provides  an 
output  in  the  form  of  information  or  an  action. 
This  is  illustrated  in  Figure  3-1.  Incoming 
data  can  originate  from  a  human  operator, 
external  sensors,  or  computer  models  which 
simulate  the  external  environment. 


The  incoming  data  is  processed  by  a  computer 
program  which  is  a  set  of  instructions  and  data 
that  were  previously  loaded  and  stored  in  the 
computer.  The  details  of  how  these  instruc¬ 
tions  and  data  are  generated  and  stored  in  the 
computer  are  discussed  later  in  this  chapter. 


INPUT  - 
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•  MODELS 
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Fig.  3-1  Computer  Definition 


Once  the  data  has  been  processed,  the  com¬ 
puter  provides  information  to  a  human 
operator;  performs  a  particular  action  such  as 
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turning  on  an  actuator  or  updating  a  data 
base;  or  provides  processed  data  to  a  com¬ 
puter  simulation  as  part  of  its  own  input  data. 

The  major  components  of  a  computer  are  the 
input  and  output  (I/O)  section,  the  central 
processing  unit  (CPU)  and  the  memory  unit 
as  depicted  in  Figure  3-2. 

3.2.1  Input/Output  Section 

For  the  computer  to  be  a  useful  device,  it  must 
be  able  to  communicate  with  people  or 
devices  outside  itself.  This  is  accomplished 
through  input  and  output  devices.  Examples 
of  input  devices  are  terminals,  keyboards,  and 
sensors  such  as  navigational  instruments,  al¬ 
timeters,  fuel  level  sensors,  and  temperature 
sensors.  Examples  of  output  devices  include 
printers,  displays,  actuators,  and  electro¬ 
mechanical  devices  that  are  part  of  the 
weapon  system.  Sometimes  the  two  functions 
are  combined  as  they  are  on  a  terminal  which 
includes  a  display  screen  and  ti  keyboard  as¬ 
sembled  into  a  single  unit.  In  short,  it  is  the 
computer  I/O  that  provides  the  interface  to 
the  rest  of  the  system. 

3.2.2  Central  Processing  Unit 

'I'he  central  processing  unit  is  the  brain  of  the 
computer.  It  is  in  the  CPU  where  the  actual 
processing  or  computations  take  place.  The 
processing  is  based  on  the  computer  program 
or  set  of  instructions  which  have  been  stored 
in  the  computer’s  memory. 

As  an  example  of  the  type  of  processing  to  be 
performed  by  a  CPU,  consider  an  aircraft 
avionics  .system  which  uses  an  inertial  plat¬ 
form  and  a  computer  program  stored  in 
memory  to  perform  the  navigational  function. 
An  inertial  platform  is  a  device  that  utilizes 
gyroscopes  to  maintain  a  fixed  attitude  with 
respect  to  some  external  reference,  usually 


the  stars,  and  accelerometers  for  measuring 
acceleration.  The  process  of  navigation  would 
involve  the  following:  Step  1:  Begin  the 
process.  This  will  require  some  housekeeping 
and  initializat’on  to  tell  the  computer  the  ini¬ 
tial  starting  position  and  the  direction  in 
which  the  platform  is  pointing.  Step  2:  Obtain 
input  data  from  the  navigational  sensors,  i.e., 
the  accelerometer  outputs  and  the  gyroscopic 
attitude  output.  Step  3:  Compute  current 
position  and  velocity  based  on  the  internally 
stored  program.  Step  4:  Output  this  informa¬ 
tion  to  the  operator  and/or  guidance  system. 
This  process  will  be  repeated  at  regular  inter¬ 
vals  to  provide  a  continuous  flow  of  naviga¬ 
tional  information.  For  the  output  to  be 
timely,  the  process  needs  to  occur  in  real¬ 
time. 

The  CPU  is  in  control  during  the  entire  time 
it  is  executing  these  instructions.  Step  3  above 
processed  the  data  received  from  the  platform 
to  produce  the  necessary  information.  In  this 
ca.se,  the  computation  involved  determining 
the  change  in  direction  relative  to  the  refer¬ 
enced  stable  platform  and  calculating  the 
velocity  by  numerically  integrating  the  ac¬ 
celeration  over  time.  By  performing  this 
proce.ss  in  real-time  and  providing  a  con¬ 
tinuous  output,  the  aircraft’s  position  and 
velocity  will  be  known  at  all  times 
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3.2.3  Memory  Unit 

The  third  major  component  of  the  computer 
is  the  memory  unit.  One  of  the  important 
aspects  of  a  computer  is  the  ability  to  store  in 
its  memory  the  instructions  and  data  required 
for  the  computer  to  perform  its  functions.  By 
storing  different  instructions  and  data,  the 
computer  can  perform  many  different  tasks 
within  the  bounds  imposed  by  the  sy.stem’s 
design  and  implementation.  In  the  previous 
example,  a  set  of  instructions  stored  in 
memory  allowed  the  computer  to  perform 
navigational  computations.  Using  a  different 
set  of  instructions,  the  computer  could  be 
used  to  determine  the  status  of  the  hardware 
components  of  the  entire  system.  This  as¬ 
sumes  that  the  appropriate  input  data  is 
provided  by  the  various  subsystems.  The  com¬ 
puter  has  the  capability  to  perform  these  func¬ 
tions  and  many  more.  This  ability  to  store  and 


execute  programs  provides  considerable 
power.  During  execution  of  a  program,  the 
computer  fetches  an  instruction  out  of 
memory,  performs  that  instruction  and  then 
steps  to  the  next  instruction.  This  continues 
until  all  the  instructions  have  been  per¬ 
formed.  The  main  or  internal  memory  is  lo¬ 
cated  within  the  computer,  but  memory  can 
also  be  located  externally  in  memory  devices 
such  as  fixed  disks,  or  tape  drives.  Internal 
memory  is  limited  in  capacity  so  external 
devices,  which  have  more  capacity,  are  used 
for  long  term  storage  of  large  programs  and 
data. 

3.2.4  Computer  Hardware 

Computer  hardware  comes  in  many  shapes 
and  sizes.  Figure  3-3  shows  Texas 
Instruments’  MIL-STD-1750A  computer, 
which  is  used  in  integrated  avionics  applica- 
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tions  such  as  the  Navy’s  Advanced  Tactical 
Aircraft  (ATA),  the  Air  Force’s  Advanced 
Tactical  Fighter  (ATF),  and  the  Army’s 
Abrams  MIMA  Tank.  About  the  size  of  a 
bread  box,  its  performance  is  comparable  to 
the  original  IBM  personal  computer. 

A  much  larger  system  is  shown  in  Figure  3-4. 
This  system  is  IBM’s  Reduced  Instruction  Set 


Architecture  (RISC)  System/6000 
FOWERstation  320  computer  and  it  is  much 
more  powerful  and  larger  than  the  MIL-STD- 
1750A  computer. 

3.3  COMPUTER  RESOURCES 

Now  that  the  computer  basics  have  been  in¬ 
troduced,  it  is  time  to  address  the  bigger  pic- 


Fig.  3-4  IBM  RISC  Systein/6000  Computer 
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ture  of  what  is  referred  to  as  computer  resour¬ 
ces.  All  too  often  important  aspects  of  a 
software  project  are  arbitrarily  relegated  to 
positions  of  insignificance  because  their  im¬ 
portance  is  not  understood  and,  therefore, 
receive  little  or  no  program  management  at¬ 
tention  until  it  is  too  late.  To  preclude  this 
from  happening,  program  managers  (PMs) 


must  be  familiar  with  till  the  components  that 
make  up  and  support  ti  computer  system.  Only 
by  fully  understanding  all  the  pieces  of  the 
puzzle,  can  PMs  properly  manage  computer 
resources.  This  doesn’t  mean  that  PMs  must 
have  detailed  knowledge  of  a  computer 
processor’s  operation  or  have  the  ability  to 
generate  software  code  for  projects.  PMs 
should,  however,  have  a  basic  understanding 
of  computer  resources  and  know  how  these 
resources  fit  into  the  overall  weapon  system 
architecture.  The  components  of  computer 
resources  are  shown  in  Figure  3-5. 

3.3.1  Embedded  Computer  Hardware 


weight,  power,  application,  or  other  technical 
considerations.  The  special  purpose  com¬ 
puter  or  processor  may  take  the  form  of  a 
"blackbox",  an  assembly  of  cards,  or  even  a 
single  card  which  is  embedded  in  the  system. 
This  means  that  the  computer  is  an  integral 
part  of  the  weapon  system. 

With  today’s  Very  High  Speed  Integrated  Cir¬ 
cuit  (VHSIC)  technology,  a  computer  can  be 
built  on  a  single  integrated  circuit,  a  piece  of 
silicon  not  much  larger  than  a  1/4  by  1/4  inch 
square.  To  the  untrained  eye,  an  embedded 
computer  system  may  be  physically  indistin¬ 
guishable  from  the  rest  of  the  system.  An 
example  would  be  a  computer  used  in  the 
flight  control  system  of  an  air-to-air  missile  or 
tne  navigational  computer  in  a  satellite. 

3.3.2  Software 

Software  is  defined  by  the  Federal  Acquisi¬ 
tion  Regulations  (FAR)  as  the  .set  of  instruc¬ 
tions  and  data  that  are  executed  in  a 
computer.  This  definition  clearly  distin¬ 
guishes  data  items,  such  as  documentation 
and  specifications  that  are  called  out  in  the 
contract,  from  the  deliverable  software  such 
as  an  operational  flight  program.  Although 
some  common  definitions  of  software  often 
include  all  the  documentation  as  well,  the 
DOD  definition  includes  only  the  executable 
form  of  the  instructions  and  data.  Software  is 
not  something  you  can  touch  or  feel.  It  is 
intangible:  it  has  no  mass,  no  volume,  no 
color,  no  odor,  no  physical  properties.  It  can 
only  be  represented  by  a  listing  or  other  forms 
of  documentation.  Software  will  be  ad- 
dre.ssed  in  greater  detail  later  in  this  chapter. 


In  weapon  systems,  the  program  manager 
needs  to  be  concerned  with  mission  critical 
computer  resources  as  defined  in  Chapter  2. 
Generally,  a  weapon  system  is  designed  with 
a  special  purpose  computer  because  of 


3.3.3  Firmware 

The  evolution  of  computer  hardware  has  also 
brought  about  the  marriage  of  hardware  and 
software  in  a  combination  called  firmware. 
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Firmware  is  defined  as  software  that  has  been 
implemented  in  hardware  using  memory 
devices  such  as  read  only  memory  (ROM), 
programmable  ROM  (PROM),  erasable 
PROM  (EPROM),  and  electrically  erasable 
PROM  devices  (EEPROM).  These  devices, 
and  other  similar  devices  which  are  generical- 
ly  referred  to  as  integrated  circuits  (ICs), 
allow  software  to  be  permanently  imple¬ 
mented  and  not  easily  changed.  In  order  to 
change  software  implemented  in  firmware, 
one  of  two  actions  must  be  taken.  If  the 
firmware  is  ROM  or  PROM,  then  these 
memories  or  ICs  must  be  physically  removed 
from  a  circuit  card  and  replaced  with  other 
ROMs  or  PROMs  that  have  been 
programmed  with  the  new  software.  If  the 
firmware  is  EPROM,  then  the  ICs  must  be 
removed,  reprogrammed,  and  reinstalled. 
The  EEPROM  can  be  altered  in  circuit  but 
this  requires  special  additional  equipment  or 
circuitry.The  EPROM  is  usually  altered  using 
an  ultra-violet  light  source  and  the  EEPROM 
can  be  altered  using  electrical  means. 

Because  of  the  difficulty  encountered  in 
changing  software  that  has  been  implemented 
in  firmware,  firmware  is  used  only  in  applica¬ 
tions  that: 

Require  Speed  -  Many  signal  processing 
applications,  such  as  electronic  warfare  sys¬ 
tems,  must  receive,  analyze,  categorize  and 
jam  radar  signals  from  hostile  threats  almost 
instantaneously.  They  cannot  tolerate  the 
relatively  slow  processing  speeds  a.ssociated 
with  general  purpose  computers.  In  these 
cases  the  various  algorithms  or  programming 
steps  are  implemented  in  firmware  in  order  to 
significantly  increase  the  processing  speed. 

Require  protection  from  unauthorized 
tampering  or  alterations  -  The  software  re¬ 
quired  to  run  a  computer  is  oftentimes  stored 
in  firmware.  By  using  these  devices,  com¬ 


puter  manufacturers  preclude  programmers 
from  inadvertently  changing  the  resident 
software,  commonly  referred  to  as  the  operat¬ 
ing  system  software,  and  possibly  causing  the 
computer  to  fail  or  to  operate  improperly. 

Require  permanent  software  -  Programs 
that  have  been  implemented  in  firmware  are 
immediately  available  in  memory  and  do  not 
have  to  be  loaded  when  the  computer  is  first 
powered  up.  This  also  provides  a  form  of 
protection  from  power  failures.  A  thoroughly 
tested  and  stable  program  is  a  good  candidate 
for  firmware. 

Firmware  introduces  an  additional  dimension 
to  software.  Because  it  is  software,  all  the 
software  configuration  management  practices 
also  apply  to  firmware.  Once  the  software  is 
implemented  in  firmware,  however,  the  ICs 
are  managed  as  hardware  configuration 
items.  In  order  to  provide  for  future  support, 
a  method  must  exist  which  traces  the  specific 
software  version  to  a  unique  piece  part. 

3.3.4  Peopleware 

People  are  also  an  important  part  of  the  sys¬ 
tem.  The  program  manager  tries  to  satisfy  the 
user’s  need  through  a  reasonable  system 
design,  but  it  is  the  user  and  support  personnel 
who  are  the  ultimate  judge  of  the  delivered 
product’s  quality.  'Phis  is  why  it  is  important 
that  the  program  manager  involve  the  user  in 
defining  requirements,  in  evaluating  test 
results,  and  interpreting  system  interface  re¬ 
quirements.  Other  chapters  will  address  the 
importance  of  involving  the  user  and  support 
personnel  in  the  development  process. 

3.3.5  Documentation 

Because  software  development  is  largely  an 
intellectual  exercise,  documentation  is  vital 
for  communicating  during  the  software 
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development  and  support  phase.  Documenta¬ 
tion  must  be  a  logical  by-product  of  the 
development  process.  As  software  develop¬ 
ment  tools  and  computer  programming  lan¬ 
guages  become  more  sophisticated, 
documentation  will  become  more  and  more 
of  an  automatic  by-product  of  the  develop¬ 
ment  process.  Until  then,  however,  the  pro¬ 
gram  manager  must  insure  that  adequate 
documentation  exists  to  accommodate 
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development  and  follow-on  support.  It  is  im¬ 
portant  to  remember  that  software  is  intan¬ 
gible,  with  no  physical  properties,  tind  that 
documentation  is  the  only  means  available  for 
describing  and  keeping  track  of  its  develop¬ 
ment  progress. 

3.3.6  Dcvelopment/Support  Facilities 

A  computer  system  consists  of  hardware, 
software,  firmware,  peopleware,  and  [)aper- 
ware  as  indicated  in  Figure  3-5.  All  these 
elements,  which  are  brought  together  in  a 
support  facility,  must  be  available  during  the 
development  and  support  phases  of  the 
weapon  system.  The  support  facility  is  an  im¬ 
portant  aspect  of  computer  resources.  It  in¬ 


cludes  not  only  the  physical  property,  such  as 
the  building,  host  computers,  and  utilities,  but 
also  the  supporting  software  documentation 
needed  for  development  and  support.  A 
software  development  facility  and  a  software 
.support  facility  are  virtually  identical  since  the 
same  software  and  hardware  tools  are  re¬ 
quired  for  both.  The  facility  may  consist  of  a 
host  computer,  which  may  be  either  a  large 
mainframe  computer  or  a  minicomputer, 
along  with  terminals  or  work  stations  for  the 
programmers,  analysts,  testers,  librarian,  and 
other  personnel  (Figure  3-6).  The  facility  may 
also  be  connected  to  other  similar  facilities 
through  local  area  networks  (LANs).  In  order 
to  perform  software  development  and  sup¬ 
port,  several  software  programs  are  required. 
These  programs  include  compilers,  linkers, 
loaders,  simulators,  editors,  and  other 
development  and  management  tools. 

3.4  COMPUTER  ARCHITECTURE 

The  computer  can  be  thought  of  as  a  collec¬ 
tion  of  hundreds  of  thousands  of  electrical 
.switches.  Each  of  these  .switches  can  be  in  one 
of  two  states,  on  or  off.  Since  the  .switch  has 
two  states,  the  status  of  any  one  "witch  can  be 
represented  by  a  "0"  or  a  "1",  i.e.,  on  =  1  and 
off  0.  The  binary  numbering  system  can  be 
used  to  represent  the  state  of  these  switches 
since  it  too  has  only  two  digits,  0  and  1 .  Instruc¬ 
tions  and  data  can  therefore  be  represented 
by  a  string  of  Os  and  Is  and  by  using  the  rules 
of  Boolean  logic,  named  after  the  English 
mathematician  and  logician  George  Boole. 
The.se  .switches  are  interconnected  to  build 
modern  electronic  computers.  Modern  com¬ 
puters,  no  matter  how  large  or  how  small, 
perform  the  following  basic  types  of  opera¬ 
tions  or  instructions: 

ARITHMETIC 

add  multiply 

subtract  tiivide 
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LOGICAL 

AND  NOT 

OR  EXCLUSIVE  OR 

TRANSFER  CONTROL 

branching  subroutines  loops 

DATA  MOVEMENT 

load  store  move 

INPLT  AND  OUTPUT 

in  out 

SYSTEM 

HALT  interrupt 

3.4.1  Bits  and  Bytes 

In  the  binary  numbering  system  a  bit  repre¬ 
sents  one  digit,  either  a  "0"  or  a  "1".  A  word  is 
a  string  of  bits  that  represent  instructions  or 
data;  the  larger  the  string  the  more  informa¬ 
tion  it  can  represent.  Any  character  can  be 
represented  by  using  "coding"  techniques. 
One  widely  used  technique  is  the  American 
Standard  Code  for  Information  Interchange 
(ASCII)  which  is  used  for  encoding  the  al¬ 
phabet,  numbers,  and  other  special  charac¬ 
ters.  There  are  128  characters  in  the  ASCII 
set,  while  another  widely  used  technique,  the 
IBM  set,  has  256  characters.  As  an  illustra¬ 
tion,  part  of  the  ASCII  alphabet  and  number 
coding  scheme  is  shown  in  Figure  3-7. 

Notice  that  the  ASCII  standard  uses  eight  bits 
or  digits  to  represent  a  character.  This  eight 
bit  word  length  is  commonly  referred  to  as  a 
BYTE  and  was  usually  the  smallest  word  size 
in  earlier  computers,  particularly  in  personal 
computers  (PCs).  The  eight  bit  structure  of 
PCs  has  been  replaced  by  the  1 6  bit  word,  with 
32  bit  structures  quickly  taking  their  place. 
Large  mainframes  have  always  used  larger 
word  sizes  such  as  32  or  64  bit  word  lengths. 
The  advantage  of  a  larger  word  length  is  that 


it  contains  more  information  in  a  single  word 
and  can  access  larger  segments  of  stored  data. 
A  computer  architecture  is  designed  around  a 
specific  word  size  since  the  internal  com¬ 
munication  between  the  CPU,  memory  and 
I/O  is  dependent  on  the  number  of  bits  in  a 


CHARACTER 

BINARY  CODE 

DECIMAL 

EQUIVALENT 

0 

CX3110000 

48 

1 

00110001 

49 

2 

00110010 

50 

3 

00110011 

51 

A 

01000001 

65 

B 

01000010 

66 

C 

01C00011 

67 

D 

01000100 

68 

E 

01000101 

69 

F 

01000110 

70 

Fig.  3-7  ASCII  Alphabet  (Partial) 


word.  For  example,  a  computer  that  has  an 
eight-bit  architecture  (8  bit  buss)  communi¬ 
cates  eight  bits  at  a  time  (in  parallel)  while  a 
16-bit  machine  communicates  16  bits  at  a 
time.  This  effectively  doubles  the  throughput. 

3,4.2  Instruction  Set  Architecture 

The  computer  architecture  and  its  internal 
logic  structure  are  designed  and  implemented 
by  the  computer  manufacturer  to  perform  a 
finite  and  fixed  set  of  instructions.  A  com¬ 
puter  with  a  minimal  set  of  instructions  can 
perform  the  same  computations  as  one  with  a 
larger  set.  The  difference,  however,  may  ap¬ 
pear  in  the  execution  time  and  the  sequence 
of  instructions  in  the  software  program.  Let’s 
assume  that  a  programmer  is  required  to 
generate  a  computer  program  to  perform  a 
particular  task.  A  program  written  for  a 
machine  with  a  large  set  of  instructions  will 
usually  require  fewer  lines  of  machine  in¬ 
structions  than  a  program  written  for  a 
machine  with  a  smaller  set  of  instructions. 
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The  difference  in  actual  machine  instruction 
sets  is  dependent  on  the  manufacturer’s  ob¬ 
jectives  in  design.  Computers  can  be 
designed  and  optimized  for  specific  applica¬ 
tions.  Some  computers  are  designed  to  per¬ 
form  very  rapid  mathematical  computations; 
others  are  designed  to  manipulate  large 
amounts  of  data  in  a  ver>'  efficient  manner; 
and  still  others  are  designed  with  a  very 
powerful  graphics  capability.  No  computer, 
however,  can  be  built  so  that  it  can  perform 
equally  well  in  all  applications.  There  is  no 
industry  standard  for  computer  design  and 
each  manufacturer  is  free  to  design  and  target 
its  machine  for  the  application  of  its  own 
choosing.  This  means  that  each  computer  has 
its  own  internal  and  fixed  repertoire  of  in¬ 
structions.  This  fixed  set  of  instructions  is 
called  the  computer’s  instruction  set  architec¬ 
ture  (ISA)  and  in  order  to  execute  a  computer 
program  on  a  particular  machine,  that  pro¬ 
gram  must  be  specifically  targeted  or  written 
for  that  machine’s  ISA.  In  other  words,  the 
binary  instructions  and  data  that  make  up  a 
software  program  are  different  for  computers 
with  different  ISAs.  Instruction  set  architec¬ 
tures  are  the  "blueprints"  that  describes  the 
interface  to  the  set  of  electronic  hardware  or 
circuitry  to  execute  the  different  types  of 
operations  or  instructions. 

Word  size  is  an  important  part  of  the  com¬ 
puter  architecture.  Recall  the  earlier  discus¬ 
sion  on  the  communication  of  instructions 
and  data  within  the  machine.  Part  of  the  basic 
design  is  determining  the  internal  signal  com¬ 
munication  paths.  This  internal  communica¬ 
tion  is  accomplished  through  the  use  of  an 
electronic  component  called  a  buss.  The  buss 
provides  parallel  signal  paths  between  the 
CPU,  memory,  and  external  devices.  A  com¬ 
puter  will  typically  have  two  busses  as  shown 
in  Figure  3-8.  The  design  architecture  will 
also  determine  the  internal  communication 
within  a  computer.  A  computer  that  has  a  16- 


bit  architecture  communicates  internally  16 
bits  at  a  time  (in  parallel).  A  computer  that 
has  a  32  bit  architecture  communicates  32  bits 
at  a  time.  This  effectively  gives  the  computer 
with  32  bits  a  greater  throughput  or  faster 
execution  capability. 

Although  there  are  no  official  standards  for 
commercial  hardware  designs  and  computer 
architectures,  the  surge  of  sales  in  personal 
computers  has  made  de  facto  standards  of 
some  Intel  and  Motorola  computer  architec¬ 
tures.  Within  the  DOD,  however,  there  does 
exist  a  standard  for  ISA,  namely  MIL-STD- 
1750A.  This  standard  has  been  used  success¬ 
fully  in  both  Air  Force  and  Navy  programs  but 
its  application  is  usually  limited  to  airborne 
and  embedded  computer  applications.  This  is 
because  this  ISA  was  initially  designed  by  the 
Air  Force  in  the  1970s  around  a  16-bit  word 
size.  It  was  specifically  intended  for  airborne 
applications  and  it  has  a  limited  memory 
capacity.  Today’s  rapidly  evolving  computer 
technology  is  quickly  making  MIL-STD- 
1750A  ISA  obsolete.  The  advantage  of 
standard  ISAs  is  portability  of  executable 
software. 

3.5  SOFTWARE  LANGUAGES 


Software  languages  are  the  vocabulary  or  lex¬ 
icon  used  to  instruct  computers  on  the  func- 


Fig.  3-8  Computer  Busses 


3-9 


Ciiapter  3  Technical  Foundations 


tions  they  will  perform.  Software  languages 
can  be  categorized  into  four  groups:  machine 
languages,  assembly  languages,  higher  order 
languages,  and  application  generators. 
Software  languages  are  also  referred  to  by 
generation,  with  machine  language  being  the 
first  generation  and  application  generators 
being  the  fourth  generation  languages 
(4GLs). 

3.5.1  Machine  Language 

Machine  language  is  the  most  primitive  and 
basic  of  all  the  languages  and  the  only  lan¬ 
guage  that  can  be  used  in  a  computer.  It  is 
written  in  binary  code  and  provides  the 
machine  the  instructions  it  is  to  execute.  The 
binary  coded  words  are  those  words  that  were 
designed  for  the  machine’s  ISA.  Program¬ 
ming  in  machine  language,  forces  the 
programmer  to  structure  the  problem  solving 
steps  in  the  same  way  the  machine  will  ex¬ 
ecute  them.  When  computers  were  first  in¬ 
troduced,  that  was  exactly  how  engineers 
constructed  their  computer  programs.  Since 
data,  instructions,  and  memory  locations  are 
represented  by  Os  and  Is  (See  Figure  3-9),  this 
method  is  very  tedious  and  error  prone  and 
becomes  nearly  impossible  for  practical 


problems. 

In  addition. 

because 

every  com- 

11010000 

00001100 

00000101 

1 1000000 

01010000 

11010000 

11000000 

00111110 

01000001 

11010000 

11000000 

00111010 

01011000 

01100000 

11000000 

10000010 

01000001 

10110000 

oooooooo 

oooooooo 

01000001 

00110000 

oooooooo 

10000110 

01011010 

10110011 

oooooooo 

oooooooo 

01000001 

00110011 

oooooooo 

OOOOOlOO 

01000110 

011000CO 

11000000 

00010100 

01011100 

10100000 

11000000 

10100010 

01011101 

10100000 

11000000 

10000010 

11100001 

01100000 

oooooooo 

OOOOOOOO 

0^01 1000 

11010000 

11000000 

00111110 

10011000 

11101100 

11010000 

00001100 

00000111 

11111110 

00000000 

00000000 

oooooooo 

00010000 

00000000 

00000000 

oooooooo 

00100000 

00000000 

oooooooo 

oooooooo 

00000110 

00000000 

00000000 

oooooooo 

00000001 

Fig.  3-9  Binary  Object  Code 


puter  has  its  own  unique  machine  language 
and,  because  of  this,  machine  language 
programs  are  not  transferable  between  dif¬ 
ferent  type  machines. 

3.5.2  Assembly  Language 

Early  in  the  history  of  computer  development, 
engineers  learned  to  use  the  power  of  the 
computer  to  assist  in  the  programming 
process.  Instead  of  directly  using  binary  code, 
the  engineer  developed  a  shorthand  notation- 
al  language  that  was  easier  to  understand. 
This  notational  language  was  called  assembly 
language.  Assembly  language  represents  each 
instruction  with  a  mnemonic  expression  and 
data  is  represented  by  its  equivalent  decimal 
number.  The  engineer  or  programmer  still 
structures  the  problem  solving  steps  the  same 
way  the  machine  executes  them;  but  now  the 
computer  itself  is  used  to  perform  the  trans¬ 
lation  from  assembly  language  into  machine 
language.  For  example,  the  assembly  lan¬ 
guage  program  in  Figure  3-10  will  find  the 
average  of  "N"  number  of  grades. 

This  program  now  needs  to  be  translated  into 
the  binary  code  that  the  machine  can  execute. 
This  translation  process,  known  as  "assembly", 
is  performed  by  another  program  called  an 
"assembler".  The  program  written  in  assembly 
language  is  known  as  the  "source"  program 
and  the  binary  code  created  by  the  assembler 
is  called  the  "object"  program  or  code.  The 
assembly  language  program  (e.g.,  the  source 
program  in  Figure  3-10)  would  then  become 
the  object  program  shown  in  Figure  3-9. 

The  introduction  of  assembly  languages 
greatly  simplified  computer  programming 
and  resulted  in  an  increase  in  productivity. 
Use  of  assembly  language,  however,  does  in¬ 
troduce  some  inefficiency  in  execution  be¬ 
cause  the  translation  process  introduces  some 
overhead  (additional  code).  Tlie  general  ap- 
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STMT 

SOURCE  STATEMENT 

1  AVERAGE 

CSECT 

1 

2 

STM 

14,12,12(13)  ! 

3  ‘THIS  PROGRAM  FINDS  THE  AVERAGE  OF  N  INTEGER  VALUES* 

4 

BALR 

12,8 

5 

USING 

*,12 

6 

ST 

13,SAVE+14 

7 

LA 

13,SAVE 

8  ‘STANDARD  LINKAGE  FROM  OPERATING  SYSTEM* 

9 

L 

6,N  ‘REGISTER  USED  TO  INCREMENT* 

10 

LA 

1 1 ,0  ‘REGISTER  1 1  USED  FOR  SUMMING* 

11 

U 

2,SSE  ‘ADDRESS  OF  FIRST  NUMBER* 

12  LOOP 

A 

1 1 ,0(3)  ‘SUM  =  SUM  +  NEXT  NUMBER* 

13 

U 

3,4,  (3)  ‘GET  ADDRESS  OF  NEXT  NUMBER* 

14 

BCT 

6,  LOOP 

15 

M 

10,  =  D'1 '  ‘EXTEND  SIGN  BIT  TO  HIGH  ORDER* 

16* 

•REGISTER  PAIR* 

17 

D 

10,N  ‘INTEGER  PART  OF  AVERAGE  IS* 

18* 

‘PUCED  IN  REGISTER  1 1  (HEX)* 

19 

‘AND  REMAINDER  IN  REGISTER  1 1  * 

20 

XDUMP 

21 

L 

13,SAVE  +  4 

22 

LM 

14,12,12,(13) 

23 

BR 

14 

24  SAVE 

DS 

18F 

25  N 

DC 

F'6' 

26  ADDR 

DC 

F' 16,32,442,988,-26,388’ 

27  END 

28 

=  F'1’D 

Fig.  3-10  Assembly  Language  Program 


proach  to  translation  employed  by  an  as¬ 
sembler  can  not  always  optimize  the  binary 
instructions  to  be  as  efficient  as  binary  code 
written  directly  by  a  good  programmer.  Be¬ 
cause  the  assembly  language  source  program 
makes  it  easier  to  understand  and  to  correct 
errors,  modifying  programs  becomes  easier 
and  the  benefits  accrued  from  this  far  out¬ 
weigh  the  inefficiency  introduced  by  the 
translation  process. 

Assembly  language,  however,  retains  an  al¬ 
most  one  for  one  correspondence  with  the 
instruction  set  of  a  particular  machine.  This 
means  that  every  machine  instruction  set  has 
a  different  assembly  language.  When 
programming  in  assembly  language,  the 
programmer  has  to  tailor  the  problem  solving 
steps  to  the  particular  machine’s  repertoire  of 
instructions.  Tliis  dictates  a  unique  assembly 
language  for  every  machine.  Because  of  this 
uniqueness,  assembly  language  programs  are 


not  transferrable  among  different  types  of 
machines. 

3.53  High  Order  Language 

The  next  advance  in  programming  occurred 
with  the  introduction  of  higher  order  lan¬ 
guages  (HOLs).  HOLs  use  statements  that 
are  more  English-like,  easier  to  understand, 
more  productive,  easier  to  support,  and  less 
dependent  on  the  computer  design.  Ex¬ 
amples  of  higher  order  languages  are 
COBOL,  FORTRAN,  Pascal,  and  Ada.  The 
CQmmon  Business  Oriented  Language 
(COBOL)  was  created  in  order  to  help  the 
business  community  manage  large  amounts  of 
data.  FORTRAN,  which  is  an  acronym  for 
FORmula  TRANslator.  was  developed  for 
scientists  and  engineers  and  allowed  the  crea- 
tion  of  mathematical  algorithms  and 
programs  without  the  need  to  know  the  details 
of  a  particular  computer  ISA.  Although  these 
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languages  can  be  used  in  applications  other 
than  those  they  were  designed  for,  they  have 
found  their  broadest  application  in  the 
domain  they  are  best  suited  for,  namely  scien¬ 
tific  and  business  applications  respectively. 
Different  problem  domains  have  created  the 
need  for  new  languages  tailored  to  the 
peculiarities  of  that  problem  solving  process. 
As  a  result  of  this  need,  hundreds  of  HOLs 
have  been  developed. 

As  with  assembly  language,  a  higher  order 
language  must  be  translated  into  a  particular 
machine  code.  This  process  of  translation  is 
called  "compiling"  and  the  translator  is  a 
software  program  called  a  compiler.  A  com¬ 
piler  is  generally  a  large  and  complex  program 
that  translates  the  HOL  source  program  to 
the  machine  executable  object  program.  Be¬ 
cause  the  object  program  is  machine  depend¬ 
ent,  the  compiler  translation  is  also  machine 
dependent.  That  means  that  each  different 
computer  must  have  its  own  unique  compiler. 

Compilers  are  typically  designed  with  the 
flexibility  for  translating  HOL  source  code  to 
many  different  machines.  This  is  done  by 
designing  the  compiler  with  a  front  end  and  a 
back  end.  When  the  translation  is  being  per¬ 
formed,  the  source  program  instructions  are 
first  processed  by  the  front  end  to  create  a 
program  in  a  generic  intermediate  assembly¬ 
like  language.  This  front  end  process  is  the 
most  difficult  part  of  the  translation.  The 
second  step  is  to  process  this  intermediate 
program  with  the  back  end  of  the  compiler. 
The  purpose  of  the  back  end,  or  code  gener¬ 
ator,  is  to  translate  the  intermediate  code  into 
the  machine  language  of  the  "target"  com¬ 
puter  (the  one  that  will  actually  execute  the 
program).  The  code  generation  process  of  a 
compiler  is  similar  to  an  assembler.  To  trans¬ 
late  to  different  target  computers,  the 
developer  has  to  build  the  unique  code  gen¬ 
erator  for  that  particular  target  computer  and 


the  front  end,  which  is  the  toughest  to 
develop,  remains  unchanged.  In  industry, 
once  the  first  compiler  is  developed,  sub¬ 
sequent  compilers  for  different  machines  can 
be  quickly  constructed.  A  partial  Ada  pro¬ 
gram  for  the  previous  programs  shown  in  Fig¬ 
ure  3-9  and  Figure  3-10  is  found  in  Figure 
3-11. 

3.5.4  Application  Generators  (4GL) 

All  the  earlier  languages,  including  the  high- 
order  languages,  are  classified  as  "procedural" 
languages.  The  user  must  define  in  detail  the 
sequence  of  operations  required  to  solve  a 
particular  programming  problem.  A  more 
recent  development  in  programming  technol- 


with  TEXTJO;  use  TEXTJO; 

with  GRADE^PACKAGE;  use  GRADE^PACKAGE; 

pfocedure  AVeRAGE_GRADES  is 

NEXT..GRADE  :  GRADE_PACKAGE.GRADE: 

THIS_HISTORY  :  GRADE_PACKAGE.GRADE_  HISTORY; 

GRADE  COUNT  :  NATURAL  ;  =»  0;  *•  number  of  grades  to  enter 

package  GRADEJO  Is  new  INTEGERJO  (GRADE_PACKAGE.GRADE): 
package  NATURAL^IO  is  new  iNTEGER_IO  (NATURAL); 

begin 

put  (’Enter  the  number  of  grades:  *); 

NATURALJO.gel  (GRADE_COUNT|; 

while  GRADE  COUNT  >  0  loop 
begin 

put  (’Enter  grade;  *); 

GRADE JO.get  (NEXT  GRADE); 
skipjine; 

GRADE^PACKAGE.ADD  (THIS  HISTORY.  NEXT  GRADE); 
GRADE^COUNT  GRADE  COUNT -1; 

exception 

when  DATA  ERROR  =  > 

put_line  (’Not  a  valid  grade;  please  enter  only  INTEGERS  ’ 

&  ’between’  &  INTEGER  'image  (GRADE'first) 

&  &  iNTEGER'image  (GRADETast)); 

skip  line; 
end; 

end  loop;  --finished  getting  grades 
put  (’The  Average  is:’); 

GRADE_IO.put  (GRADE  PACKAGE.AVERAGEfTHIS  HISTORY)); 
newjine; 


Fig.  3-11  Ada  Program 
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ogy,  called  application  generators  or  fourth- 
generation  languages  (4GLs),  can  greatly  in¬ 
crease  programmer  productivity.  A  4GLs 
allows  an  end  user  to  build  applications 
programs  without  resorting  to  coding  the 
details  in  a  particular  computer  language. 
Since  most  application  generators  are  non¬ 
procedural,  they  allow  the  programmer  or 
user  to  define  the  problem  while  the  system 
defines  the  steps  required  to  solve  the  prob¬ 
lem. 

To  make  the  distinction  between  the  proce¬ 
dural  and  nonprocedural  languages  clearer, 
one  may  use  the  analogy  of  providing  instruc¬ 
tions  to  a  taxi  driver.  With  the  procedural 
language  one  must  tell  the  driver  exactly  what 
to  do.  "Drive  500  yards  due  North.  Turn  left. 
Drive  380  yards,  etc."  With  a  nonprocedural 
language  one  may  say,  "Take  me  to  the 
Criterion  Theater  on  Main  Street."  The  most 
powerful  nonprocedural  languages  may  per¬ 
mit  one  to  simply  say,  'Take  me  to  see  Gone 
With  the  Wind."  The  taxi  driver  will  then 
search  theater  listings  to  determine  where  the 
movie  is  playing.  In  this  case  the  taxi  driver 
might  say  that  the  request  is  illogical,  if  the 
nearest  theater  at  which  this  movie  is  playing 
is  in  another  city  or  state. 

Fourth-generation  languages  vary  greatly  in 
their  power  and  capabilities.  While  some 
fourth-generation  languages  can  be  used  to 
create  complete  applications,  many  are  report 
generators  or  graphics  packages.  Most  are 
dependent  on  a  well-defined  data  base,  and 
are  designed  for  only  a  specific  class  or  range 
of  applications.  Vendors  of  the  more  com¬ 
prehensive  products  describe  their  more 
limited  competition  as  "not  true  fourth- 
generation  languages."  However,  it  is  often 
better  to  use  a  language  designed  for  a  limited 
set  of  functions  because  it  is  easier  to  learn 
than  a  full  programming  language.  Lotus  1-2- 
3  is  a  good  example  of  this  kind  of  language. 


This  language  has  gained  a  vast  number  of 
users  because  it  made  it  easy  to  manipulate 
spreadsheet  data  [5]. 

Fourth-generation  languages  can  be  very 
beneficial  in  developing  mission  support 
software;  however  at  the  present  time  there 
are  few  languages  targeted  toward  the  em¬ 
bedded  computer  market.  These  languages, 
in  general,  tend  to  develop  machine  language 
that  is  even  less  efficient  than  the  machine 
language  developed  by  HOLs,  which 
mitigates  against  using  them  to  develop  real¬ 
time  systems. 

3.6  ADA 

For  most  DOD  applications,  policy  requires 
that  developers  use  the  Ada  programming 
language.  This  policy  will  be  discussed  in 
more  detail  in  Chapter  4;  however,  because  of 
the  widespread  implications  of  the  policy,  the 
program  manager  needs  to  know  more  about 
the  language  and  its  supporting  environment. 

3.6.1  Ada  Design 

The  Ada  programming  language  was 
designed  with  three  overriding  concerns:  pro¬ 
gram  reliability  and  support,  programming  as 
a  human  activity,  and  efficiency  [1].  The  result 
of  these  concerns  was  the  creation  of  a  lan¬ 
guage  that  embodies  the  principles  of  modern 
software  engineering  practices.  Applying 
these  principles,  or  tools  of  the  trade,  helps 
one  deal  with  two  very  real  and  difficult 
aspects  of  large-scale  software  developments: 
complexity  and  change. 

Ada  was  specifically  designed  to  encourage  or 
help  the  engineer  or  programmer  develop 
software  that  is  reliable  and  simple  to  main¬ 
tain.  In  particular,  Ada  emphasizes  program 
readability  over  ease  of  writing.  For  example, 
the  rules  of  the  language  require  that  program 
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variables  be  explicitly  declared  a  particular 
type  such  as  real,  integer,  Boolean,  etc.  Since 
the  type  of  a  variable  is  unalterable,  compilers 
can  ensure  that  the  only  operations  per¬ 
formed  on  variables  are  those  allowed  by  the 
rules  of  the  Ada  language  syntax.  This 
prevents  the  programmer  from  attempting  to 
perform  an  illegal  operation  such  as  multiply¬ 
ing  an  integer  value  with  a  string  variable. 
Furthermore,  error-prone  notations  have 
been  eliminated  because  the  language  avoids 
the  use  of  cryptic  encoded  forms  in  favor  of 
more  English-like  constructs.  Finally,  the 
language  offers  support  for  separate  compila¬ 
tion  of  program  units  in  a  way  that  facilitates 
program  development  and  support. 

A  major  strength  of  Ada  lies  in  its  ability  to 
perform  numerous  checks  both  during  com¬ 
pilation  and  at  run  time.  The  richness  of  the 
syntax  makes  it  more  difficult  for  a  program¬ 
mer  to  code  and  compile  a  program,  but  once 
compiled  an  Ada  program  is  more  apt  to  run 
correctly  than  a  program  written  in  any  other 
language.  In  other  words,  Ada  enforces  a  cer¬ 
tain  amount  of  discipline  into  the  program¬ 
ming  process.  This  is  a  significant 
contribution  to  reliable  software. 

Concern  for  the  human  programmer  was  also 
stre.ssed  during  the  language  design.  An  at¬ 
tempt  was  made  to  keep  the  language  as  small 
as  possible  by  trying  to  avoid  the  pitfalls  of 
excessive  complexity.  Ada  uses  simpler 
designs  to  provide  language  constructs  that 
correspond  intuitively  to  what  the  users  would 
expect. 

Like  many  other  human  activities,  software 
programs  are  becoming  larger  and  their 
development  is  becoming  ever  more 
decentralized  and  distributed.  Consequently, 
the  Ada  designers  provided  the  ability  to  as¬ 
semble  a  program  from  independently 
produced  software  components. 


3.6.2  Ada  Compilers 

No  high  order  language  can  avoid  the  address¬ 
ing  the  issue  of  compiler  translation  efficien¬ 
cy,  because  a  poor  compiler  can  result  in  poor 
run-time  efficiency  and  inefficient  use  of 
storage  affecting  all  machines  and  all 
programs.  During  Ada  specification  develop¬ 
ment,  every  proposed  construct  of  the  lan¬ 
guage  was  examined  in  the  light  of  current 
state-of-the-art  compiler  implementation 
technique*;.  Any  proposed  construct  whose 
implementation  was  unclear  or  that  required 
excessive  machine  resources  was  rejected  [2], 
However,  there  are  substantial  differences 
among  validated  Ada  compilers.  To  explain 
the  reasons  for  these  differences,  it  is  ap¬ 
propriate  to  explain  what  validation  means. 
Validation  is  the  official  DoD  process  used  to 
demonstrate  that  a  candidate  Ada  compiler 
conforms  to  the  Ada  language  standards. 
Relevant  policy  for  validation  is  documented 
in  the  Ada  Compiler  Validation  Procedures, 
Version  2,  published  by  the  AJPO,  May  1989. 
This  policy  states  that  Ada  compilers  must  be 
validated  by  the  Ada  Validation  Facility 
(AVF)  managed  by  the  Ada  Validation  Or¬ 
ganization  (AVO).  A  compiler  will  receive 
certification  if  it  passes  all  applicable  tests  in 
the  Ada  Compiler  Validation  Capability 
(ACVC)  test  suite.  The  test  suite,  which  is 
now  updated  every  eighteen  months,  encom- 
pas.ses  six  different  classes  of  test  and  includes 
more  than  four  thousand  individual  tests. 
While  the  testing  is  extensive,  the  individual 
tests  tend  to  test  simple  capabilities  rather 
than  capacity  or  synergistic  issues.  Validation 
does  not  guarantee  that  the  compiler  is  error 
free,  nor  does  it  imply  anything  about  the 
performance  or  functionality  of  the  compiler. 

Nelson  H.  Weiderman  of  SEI  states  that 
"There  is  an  important  difference  between 
validation  and  evaluation.  Validation  te.sts 
conformance  to  ANSIIMIL-STD-181 5A. 
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Validation  cannot  he  relied  upon  for  determin¬ 
ing  whether  a  given  Ada  compiler  is  fit  for  use 
in  a  particular  application."  Evaluation  issues 
are  covered  in  the  Ada  Adoption  Handbook: 
Compiler  Evaluation  and  Selection  [5]. 

3.63  Ada  Language  Features 

Ada  has  incorporated  the  following  software 
engineering  features: 

Abstraction  is  a  simplified  description  or 
specification  that  suppresses  some  of  the 
details  or  properties.  A  good  abstraction  is 
one  that  emphasizes  data  that  is  significant 
and  suppresses  immaterial  details.  This 
facilitates  the  management  of  complexity  in 
software  development  and  maintenance.  For 
example,  a  vehicle  is  an  abstraction  at  a 
general  level.  One  can  provide  further  detail 
to  the  class  of  vehicles  by  identifying  an 
automobile.  Then  one  can  further  describe 
the  various  characteristics  of  an  automobile 
such  as  a  six  cylinder  engine,  two  door  sedan 
and  so  on  to  describe  a  specific  automobile. 
This  of  course  can  continue  down  to  the  smal¬ 
lest  component  necessary.  Notice  that  each 
lower  level  adds  detail. 

Information  hiding  is  the  technique  of 
providing  only  the  essential  information 
necessary  for  interfacing  with  a  given  unit.  A 
specification  control  drawing  is  an  example  of 
information  hiding.  This  type  of  drawing 
provides  information  on  the  inputs  and  out¬ 
puts  of  a  particular  device  without  providing 
any  details  (  n  the  internal  structure.  This 
prevents  detailed  information  from  confusing 
essential  information.  Abstraction  and  infor¬ 
mation  hiding  assist  programmers  deal  with 
complexity. 

Modularity  is  the  principle  of  logical  structur¬ 
ing.  One  decomposes  a  design  through  levels 
of  abstraction  so  that  it  has  the  properties  of 


loose  coupling  and  tight  cohesion.  A  module 
can  be  describ ’d  as  an  entity  or  unit  whose 
internal  elements  are  tightly  bound  or  related 
(cohesion)  but  with  light  interconnections 
(coupling).  In  design,  one  would  collect  all 
logically  related  resources  into  one  niodule. 
This  is  the  principle  of  localization. 
Modularity  and  localization  help  the 
programmer  to  deal  with  complexity  as  well 
as  with  change.  The  effects  of  change  can  be 
better  controlled  or  isolated  through  modular 
design  and  localization. 

All  these  features  contribute  to  reliable, 
maintainable,  understandable,  and  efficient 
designs.  They  also  allow  the  programmer  to 
cope  with  the  complexities  of  large-scale  sys¬ 
tems  and  the  inevitability  of  change.  Ada  in¬ 
corporates  all  of  these  attributes.  The 
different  concepts  embraced  by  Ada  are  not 
new  to  programming  languages.  What  is  uni¬ 
que  is  that  Ada  is  the  first  language  to  com¬ 
bine  all  these  features  into  a  single  language. 

There  are  some  unique  features  provided  by 
Ada  that  are  not  provided  by  any  other 
production  languages:  the  package,  exception 
handling,  tasking,  and  generics. 

Packages  are  entities  or  collections  of  related 
objects  and  their  operations.  This  collection 
of  resources  can  be  viewed  as  a  wall  surround¬ 
ing  a  collection  of  logically  related  entities 
such  as  operations,  data  types,  and  related 
program  units  [2].  The  package  enforces  the 
principles  of  modularity,  localization, 
abstraction,  and  information  hiding. 

Exception  handling  provides  a  controlled  way 
to  exit  from  an  abnormal  event.  In  real-time 
operations,  one  cannot  allow  an  abnormal 
event,  such  as  division  by  zero  or  a  register 
overflow,  to  halt  the  entire  process.  Through 
exception  handling,  abnormal  events  are 
flagged  during  processing  and  purposefully 
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handled  to  prevent  a  catastrophic  failure. 
This  is  done  in  one  of  three  ways:  allowing 
execution  to  follow  an  alternate  path,  restart¬ 
ing  the  operation  at  a  controlled  point,  or 
overriding  the  current  data  with  default 
values.  Exception  handling  is  one  way  of 
designing  graceful  degradation  into  the 
software. 

Tasking  is  very  important  in  real  time  opera¬ 
tions.  Tasking  allows  concurrent  or  parallel 
processing  to  occur  in  the  same  or  separate 
processors.  In  the  real  world,  processes  are 
generally  concurrent.  This  approach  to 
design  breaks  the  sequential  mindset  but  is 
one  of  the  features  of  Ada  that  is  most  often 
criticized.  Real-time  applications  require 
fast,  reliable  completion  of  tasks  whose 
priorities  are  constantly  changing;  however, 
Ada  completes  each  task  before  starting 
another  task  and  Ada  fixes  priorities  at  com¬ 
pilation  time  and  is  therefore  inflexible  to  the 
changing  environment  or  the  real  world. 
Work-arounds  are  currently  available  to  cir¬ 
cumvent  this  problem,  and  this  weakness  will 
be  one  of  the  areas  addressed  when  the  lan¬ 
guage  specifications  arc  updated. 

Generics  is  a  feature  that  reduces  complexity 
and  encourages  the  production  of  reusable 
code  components.  The  concept  of  generics  is 
similar  to  a  template  whereby  a  structure  and 
associated  operations  are  defined  for  later  u.se 
in  specific  application.  For  example,  a 
generic  routine  can  be  created  for  sorting  any 
number  of  items.  The  process  of  sorting  is 
the  same  whether  one  is  sorting  numbers, 
names,  or  objects.  In  other  languages,  one 
creates  different  sort  routines  for  each  ap¬ 
plication.  With  the  generic  capability  in  Ada, 
one  routine  will  suffice.  This  reduces  the  need 
for  multiple  programs  for  performing  the 
same  task.  It  also  aids  in  dealing  with  large 
complex  systems  by  introducing  common 
generic  packages. 


It  is  important  to  understand  that  Ada 
provides  a  means  for  achieving  good  designs 
because  it  embodies  principles  of  good  sys¬ 
tems  and  software  engineering.  It  is  also  im¬ 
portant  to  know  that  the  Ada  programming 
language  itself  is  only  a  small  part  of  the 
process  of  designing  sound  systems.  The 
design  and  development  of  good  software  re¬ 
quires  more  than  just  a  good  programming 
language.  One  must  still  employ  sound  design 
practices  and  procedures,  strong  configura¬ 
tion  management  practices,  and  good  .system 
design  tools  and  aids.  In  addition,  all  program¬ 
mers  must  be  adequately  trained  on  the  use  of 
these  methods  and  tools. 

3.6.4  Ada  Environment 

In  the  past,  software  tools  were  as  diverse  and 
hardware  unique  as  the  languages  they  sup¬ 
ported.  The  goal  of  the  early  developers  of 
Ada  was  to  develop  a  total  environment  which 
supported  the  systems  development  life  cycle. 
To  achieve  portability,  the  environment  is 
broken  into  the  Kernel  Ada  Programming 
Support  Environment  and  the  Ada  Progrtim- 
ming  Support  Environment. 

3.6.4.1  Kernel  Ada  Programming  Support 
Environment  (KAPSE) 

I’he  KAPSE  contains  all  low-level  features 
nece.s.sary  to  rehost  onto  another  .system.  It 
akso  supports:  database  access,  input/output, 
terminal  to  tool  access,  and  the  runtime  sys¬ 
tem 

3.6.4.2  Ada  Programming  Support  Environ¬ 
ment  (APSE) 

The  APSE  typically  consists  of: 

Editor.  An  interactive  tool,  preferably  one 
that  is  language  specific  for  creating 
documentation  and  source  code. 
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Debugger.  A  debugger  is  a  programmer 
productivity  tool  for  finding  errors.  It  should 
be  integrated  with  the  compiler  to  provide 
source  code  information. 

Linker.  The  linker  joins  together  object 
modules  into  executable  modules  which  will 
run  on  the  target  machine. 

Configuration  Manager.  This  tool  is  used  to 
keep  track  of  the  different  versions  of  the 
documentation  and  code. 

Static  Analyzer.  This  tool  provides  informa¬ 
tion  such  as  number  of  lines  of  code,  number 
of  comments,  level  of  complexity,  and 
measures  of  coding  style. 

Dynamic  Analyzer.  This  tool  monitors  and 
reports  the  execution  behavior  of  object  code 
so  that  the  programmer  can  "fine  tune"  the 
system. 

Library  Management  Tools.  These  are  tools 
which  automatically  recompile  and  relink  ob¬ 
ject  modules. 

Test  Tools.  Tools  to  automatically  generate 
test  data  and  compare  actual  output  to  ex¬ 
pected  output. 

3.6,4.3  Ada  Program  Design  Language 

One  of  the  most  important  tools  currently 
used  for  designing  software  is  a  program 
design  language  (PDL).  The  DOD’s  policy 
concerning  PDLs  is  given  in  DOD  Directive 
34(i5.2,  Use  ofAtla  in  Weapon  Systems  which 
states: 

"An  Ada-based  program  design  language  shall 
be  used  during  the  designing  of  software.  Use  of 
a  PDL  that  can  be  successfully  compiled  by  a 
validated  Ada  compiler  is  encouraged  in  order 
to  facilitate  the  ponability  of  the  design." 


A  PDL  is  a  formal  language  (sometimes 
referred  to  as  pseudo-code)  for  specifying  the 
"blueprint"  for  software  implementation. 
When  Ada  is  used  as  a  PDL,  the  properties  of 
the  language  allow  the  "blueprint"  of  a 
software  design  to  be  run  through  an  Ada 
compiler  with  the  restriction  that  it  doesn’t 
execute.  This  allows  the  compiler  to  perform 
all  the  syntactic  and  semantic  checks  it  nor¬ 
mally  performs.  No  other  HOL,  currently  in 
use,  has  its  own  PDL.  Several  Ada-based 
PDLs  exist.  An  example  of  one  is  IEEE  Stand¬ 
ard,  IEEE-STD-99()-1987. 

A  PDL  is  not  a  panacea  for  software  develop¬ 
ment.  Poor  software  designs  can  be  produced 
using  a  PDL.  PDLs,  however,  have  major  ad¬ 
vantages  which  can  make  the  process  of 
software  design,  code,  test  and  integration  a 
less  painful  process. 

3.7  Ada  SURVIVAL 

It  is  important  to  understand  that  the  Ada 
programming  language  is  still  in  a  period  of 
transition,  and  although  the  risk  of  using  Ada 
is  much  less  than  it  was  only  a  few  years  ago, 
there  are  still  several  factors  to  consider  when 
evaluating  the  risks  of  using  Ada.  The 
Software  Engineering  Institute’s  publication 
Qnth\ed  Ada  Adoption  Handbook:  A  Program 
Manager’s  Guide  [4]  provides  considerable 
detail  and  suggestions  for  dealing  with  Ada  in 
today’s  environment.  Figure  3-12  contains  a 
checklist  of  helpful  hints  for  those  embarking 

•  Get  training  and  experience 

-  Be  aware  of  Ada's  benefits  and  shortcoming 

-  Select  a  contractor  with  a  proven  track  record 

-  Select  a  compiler  based  on: 

characteristics  of  your  application 
-  evaluation  of  compiler  performance 
--  evaluation  of  vendor  support  services 

-  Invest  in  system  support  tools 

-  Know  your  risk  and  manage  it 


Fig.  3-12  Ada  Survival  Checklist 
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tern  development.  One  must  understand  that 
work-arounds  exist  for  many  of  the  known 
deficiencies  of  Ada. 
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CHAPTER4 


SOFTWARE  ACQUISITION  POLICY 


4.1  INTRODUCTION 

This  chapter  summarizes  the  DOD’s  policies 
governing  the  acquisition  of  mission  critical 
computer  resources  (MCCR).  In  dealing  with 
policy,  it  may  be  useful  to  understand  the 
history  behind  its  implementation.  Towards 
that  end,  this  chapter  will  provide  a  historical 
perspective  of  the  various  laws,  regulations, 
and  initiatives  that  relate  to  MCCR. 

As  indicated  in  the  previous  chapter,  com¬ 
puters  and  software  have  become  an  extreme¬ 
ly  vital  component  of  a  weapon  system.  In  a 
span  of  only  twenty-five  to  thirty  years,  the 
dependence  on  software  and  cost  of  software 
has  grown  tremendously.  With  this  growth, 
there  has  been  an  accompanying  rise  in  the 
technical  and  management  problems  across 
all  of  the  services. 

4.2  BROOKS  BILL 

Prior  to  30  October  1965,  there  was  no  form 
of  standardization  or  control  over  the 
procurement  of  automatic  data  processing 


(ADP)  equipment  within  the  federal  govern¬ 
ment.  On  that  date,  however.  Public  Law  89- 
306  (otherwise  known  as  the  Brooks  Bill)  was 
signed  by  President  Johnson.  This  bill  was 
intended  to  promote  competition  and  insure 
stability  in  the  procurement  of  ADP  resour¬ 
ces.  By  1976,  36%  of  the  systems  were 
procured  in  a  fully  competitive  manner  and, 
according  to  the  General  Services  Ad¬ 
ministration  (GSA),  over  $681  million  in  cost 
avoidance  has  been  achieved  in  302  competi¬ 
tive  ADP  contracts  [1]. 

Traditionally,  computers  and  software  have 
been  viewed  by  top  level  management  as  tools 
for  improving  efficiency  and  conserving 
resources.  Although  this  is  true,  computer 
resources  need  to  be  treated  in  the  same  man¬ 
ner  as  other  acquisitions,  not  as  mere  tools. 
Although  primarily  directed  toward  the 
procurement  of  ADP  equipment,  the  Brooks 
Bill  forces  federal  agencies  to  analyze  their 
ADP  requirements,  like  they  would  for  other 
systems,  and  compete  for  the  most  economic 
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and  efficient  system.  In  an  effort  to  achieve 
this  goal,  the  Brooks  Bill  assigned  respon¬ 
sibilities  as  follows:  the  GSA  was  given  the 
authority  for  procuring  ADP  resources  re¬ 
quired  by  federal  agencies;  the  Office  of 
Management  and  Budget  (OMB)  was  to  pro¬ 
vide  policy  guidance  and  overall  leadership 
(i.e.,  they  were  to  act  as  mediator  in  resolving 
any  user  and  GSA  disputes);  and  the  National 
Institute  of  Standards  (formerly  the  National 
Bureau  of  Standards)  was  to  develop  ADP 
standards. 

'I'he  Brooks  Bill,  however,  does  not  permit  the 
GSA  to  interfere  with  an  agency’s  (user) 
determination  of  its  ADP  requirements.  The 
user  determines  its  requirements  for  ADP 
etiuipment  and  the  potential  method  of 
procurement.  The  method  of  procurement  is 
then  a[)proved  by  the  GSA  and  any  disputes 
resolved  by  the  OMB. 

The  DOD  considers  MCCR  exempt  from  the 
provisions  of  the  Brooks  Bill  because  MCCR 
is  not  specifically  addressed  in  the  ADP 
definition.  Therefore,  the  DOD  has  con¬ 
tinued  to  procure  MCCR  as  part  of  the 
weapon  system  using  major  system  acquisi¬ 
tion  guidelines.  Additional  legislation  in  the 
form  of  the  Warner-Nunn  Amendment  and 
viirious  DOD  Directives  and  Instructions  has 
further  approved  this  inter[metation. 

4.3  1)01)  DIRECTIVE  5000.29 

DOD  Directive  5()()().29,  Management  of 
Computer  Resources  in  Major  Defense  Systems 
was  published  on  26  April  1976.  Its  purpose 
was  to  establish  a  DOD  policy  for  the  manage¬ 
ment  and  control  of  computer  resources 
during  the  life  cycle  of  major  weapon  .systems. 

The  directive  was  the  first  major  step  under¬ 
taken  by  the  DOD  to  tiddress  the  growing 
software  problem.  It  represented  the 


department’s  first  formal  recognition  that 
software  is  critical  to  weapon  systems  and 
should  be  managed  as  a  configuration  item  in 
the  same  manner  as  hardware.  As  such, 
software  requirements  should  be  validated 
and  risk  analyses  performed  prior  to  a  Mile¬ 
stone  II  decision  in  order  to  insure  that  the 
software  requirements  reflect  the  operational 
requirements.  All  of  the  management  tools 
used  in  the  development  of  hardware  should 
be  applied  to  software  (e.g.,  configuration 
management,  based ne\milestone  manage¬ 
ment,  and  life  cycle  support  planning  tools). 

In  an  effort  to  help  carry  out  the  intent  of  this 
directive,  a  Management  Steering  Committee 
for  Embedded  Computer  Resources  was  also 
established  in  1976  and  its  charter  was  con¬ 
tained  in  the  directive.  The  purpose  of  the 
steering  committee  is  to  increase  the  visibility 
and  improve  the  management  of  computer 
re.sources  within  the  DOD,  to  formulate  a 
coordinated  technological  base  program  for 
software,  and  to  integrate  computer  resource 
policy  into  the  normal  process  of  major  .system 
acquisitions. 

The  steering  committee  is  composed  of  two 
boards:  the  Executive  Board  and  the  Manage¬ 
ment  Advisory  Board.  The  Executive  Board  is 
responsible  for  the  development  of  policy 
necessary  for  the  acquisition  and  manage¬ 
ment  of  computer  resources  in  major  defense 
.systems.  It  consists  of  a  representative  from 
the  Assistant  Secretary  of  Defense  (Installa¬ 
tions  and  Lxtgistics),  who  is  the  chairman,  the 
Deputy  Director  Research  &  Engineering; 
the  Director,  Telecommunications  and  Com¬ 
mand  and  Control  Systems;  the  Assistant 
Secretary  of  Defense  (Comptroller);  and  the 
Assistant  Secretary  of  Defense  (Intelligence). 

The  Management  Advisory  Board  is  respon¬ 
sible  for  coordinating  technology  efforts 
among  the  DOD  components,  for  conducting 
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policy  impact  assessment  for  the  Executive 
Board  relating  to  computer  resources,  and  for 
reviewing  computer  resource  technology 
programs  for  policy  consistency.  It  consists  of 
representatives  from  the  Navy,  Army,  Air 
Force,  Office  of  Joint  Chiefs  of  Staff,  Defense 
Communications  Agency,  National  Security 
Agency,  Defense  Advanced  Research 
Projects  Agency,  and  Deputy  Director 
(T&E). 

4.4  WARNER-NUNN  AMENDMENT 

The  Warner-Nunn  Amendment  (Section  908 
of  Public  Law  97-86,  the  DOD  Authorization 
Act,  1982)  was  implemented  in  order  to 
broaden  the  range  of  embedded  computer 
resources  excluded  from  the  provisions  of  the 
Brooks  Bill.  It  was  intended  to  provide  the 
DOD  with  more  control  over  the  acquisition 
of  computer  resources  that  are  an  integral 
part  of  weapon  systems. 

The  Warner-Nunn  Amendment  defined 
those  computer  resources  which  are  exempt 
from  the  Brooks  Bill.  It  defined  MCCR  as 
those  computer  resources  that  perform  the 
following  functions,  operation,  or  use  [2]: 

(a)  Involves  intelligence  activities; 

(b)  Involves  cryptoanalytic  activities  related 
to  national  security; 

(c)  Involves  the  command  and  control  of 
military  forces; 

(d)  Involves  equipment  that  is  an  integral  part 
of  a  weapon  system; 

(e)  Is  critical  to  the  direct  fulfillment  military 
or  inter  ^ence  missions. 

The  essential  test  as  to  whether  the  acquisi¬ 
tion  of  computer  resources  is  covered  by  the 


Warner-Nunn  Amendment  or  the  Brooks  Bill 
is  the  intended  use  of  the  equipment  and  ser¬ 
vices,  and  not  their  commercial  market  place 
availability.  Interpretation  of  these  policies 
must  not  be  lightly  rationalized  and  used  as  an 
excuse  to  depart  from  sound  business  and 
management  practices.  Where  there  is  doubt 
as  to  the  applicability,  case-by-case  deter¬ 
minations  shall  be  made  by  the  Under 
Secretary  of  Defense  (Research  and  En¬ 
gineering),  in  coordination  with  the  Assistant 
Secretary  of  Defense  (Comptroller)  [3]. 

4.5  MCCR  STANDARDIZATION 

The  explosion  in  the  number  of  weapon  sys¬ 
tem  computer  applications  that  occurred 
during  the  late  1960s  and  the  1970s,  resulted 
in  a  comparable  explosion  in  the  number  and 
types  of  computers  and  programming  lan¬ 
guages.  By  the  mid-seventies,  there  we’'e 
literally  hundreds  of  different  computer 
programming  languages  being  used  to 
generate  military  systems  software.  Along 
with  these  languages,  there  was  an  almost 
equal  number  of  different  computers  in  use. 
The  result  was  that  engineers,  technicians, 
and  computer  programmers  who  were  sup¬ 
porting  a  particular  weapon  system  could  not 
support  a  different  weapon  system  without 
costly  retraining  and  delays.  Rarely  would 
two  different  weapon  systems  use  the  same 
computer  or  the  same  programming  lan¬ 
guage.  This  led  to  needless  duplication  of 
effort  and  inefficient  use  of  human  resources. 
Since  each  system  had  its  own  computer  and 
programming  language,  each  system  was  uni¬ 
que  and  required  unique  resources. 

In  order  to  minimize  weapon  system  software 
support  costs  and  to  promote  interoperability 
between  the  various  systems,  the  DOD 
focused  on  three  areas:  higher  order  lan¬ 
guages  (HOLs),  the  software  development 
process,  and  computer  hardware.  The 
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specific  policy  and  guidance  for  each  area  are 
the  following: 

HOLs 

DOD  Directive  3405.1, 

Computer  Programming  Language  Policy 

DOD  Directive  3405.2, 

Use  of  Ada  in  Weapon  Systems 

Software  Development 

DOD-STD-2167A, 

Defense  System  Software  Development 

DOD-STD-2168, 

Defense  System  Software  Quality  Program 

Computer  Hardware 
MIL-STD-1750A, 

Mrborne  Computer  Instruction  Set  Architec¬ 
ture. 

nOLs  and  computer  hardware  were  already 
discussed  in  Chapter  3.  DOD-STD-2167A 
and  2168  are  addressed  in  Chapter  5. 

4.6  DOD  DIRECTIVE  3405.2 

Standardization  of  HOLs  has  been  an  issue 
within  the  DOD  since  the  eai  ly  1970s.  In  1976 
DOD  liistruction  5000.31,  Interim  List  of 
DOD  Approved  Higher  Order  Languages 
(HOI A  was  issued  as  an  interim  measure  to 
limit  the  number  of  DOD  approved  HOLs  to 
six: 


DOD 

FORTRAN 

COBOL 

Army 

TACPOL 

Navy 

CMS-2M 

CMS-2Y 

Air  Force  JOVIAL  J3 
JOVIAL  J73 

Later  that  instruction  was  amended  to  include 
ATLAS  as  an  approved  HOL  for  automatic 
test  equipment.  Concurrently,  the  DOD  in¬ 
itiated  a  fully  competitive  program  to  develop 
a  common,  preferred,  single  HOL  for  DOD 
software  development  programs.  The  out¬ 
come  of  these  efforts  was  the  Ada  program¬ 
ming  language.  DOD  Directive  3405.2,  Use  of 
Ada  in  Weapon  Systems,  was  published  on 
March  30,  1987.  This  directive  established 
DOD  policy  for  the  use  of  Ada  as  the  single, 
common,  HOL  in  the  development  of  com¬ 
puters  integral  to  weapon  systems.  Com¬ 
puters  are  defined  as  being  integral  to  a 
weapon  system  if  they  are: 

(a)  Physically  a  part  of,  dedicated  to,  or  essen¬ 
tial  to  the  real  time  performance  of  the  mis¬ 
sion; 

(b)  Used  for  specialized  training,  diagnostic 
test  and  maintenance,  simulation,  or  calibra¬ 
tion; 

(c)  Used  for  R  &  D  of  weapon  systems. 

The  directive  applies  to  all  new  weapon  sys¬ 
tems  entering  into  development  (prior  to  Full 
Scale  Development  (FSD))  and  to  major 
upgrades  (greater  than  1/3  modification)  to 
existing  systems.  There  are  three  exceptions, 
however,  to  using  the  Ada  programming  lan¬ 
guage: 

(a)  If,  on  the  effective  date  of  the  directive,  a 
programming  language  other  than  Ada  was 
already  in  use  during  the  FSD  phase  of  a 
weapon  system,  then  diat  particular  language 
may  continue  to  be  used  throughout  the 
deployment  and  software  support  phases  un¬ 
less  the  system  is  undergoing  a  major  software 
upgrade. 
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(b)  Ada  is  preferred,  but  not  required,  as  a  test 
language  to  be  used  solely  for  hardware 
under  test  equipment. 

(c)  Ada  is  preferred,  but  not  required,  for 
commercially  available,  off-the-shelf 
software  that  will  not  be  modified  by  the 
DOD. 

Except  for  the  conditions  stated  above,  a 
waiver  is  required.  Authority  for  issuing 
waivers  is  delegated  to  each  DOD  component 
only  on  a  specific  system  or  subsystem  basis. 
For  each  proposed  waiver,  a  full  justification 
will  be  prepared  and  will  include  analysis  of 
developmental  risk,  technical  performance, 
life  cost,  and  schedule  impact. 

4.7  DOD  DIRECTIVE  3405.1 

The  overall  policy  for  DOD  computer  lan¬ 
guages,  DOD  Directive  3405.1,  Computer 
Programming  Language  Policy,  was  published 
2  April  1987,  three  days  after  the  Ada  direc¬ 
tive.  It  superseded  DOD  Instruction  5000.31 
and  revised  the  list  of  approved  DOD  HOLs 
to  be  used  for  the  development  and  support 
of  all  computer  resources  managed  under 
DOD  Directive  5000.29  and  DOD  Directive 
7920.1  (See  Section  4.13).  The  approved  list 
of  the  DOD  programming  languages  is:  Ada, 
C/ATLAS,  COBOL,  CMS-2M,  CMS-2Y, 
FORTRAN,  JOVIAL  (J73),  MINIMAL 
BASIC  PASCAL,  and  SPI71. 

This  directive  stresses  standardization  of 
HOLs,  and  establishes  Ada  as  the  single,  com¬ 
mon  preferred  language  within  the  DOD. 
When  Ada  is  not  used,  only  the  other  ap¬ 
proved  standard  languages  listed  shall  be 
used.  This  directive  serves  to  limit  the  number 
of  HOLs  used  in  the  DOD  and  facilitates  the 
transition  to  Ada.  The  order  of  preference  is 
based  upon  life  cycle  cost  and  impact  as  fol¬ 
lows: 


(a)  Off-the-shelf  applications  packages  and 
advanced  software  technology; 

(b)  Ada-based  software  and  tools; 

(c)  Approved  standard  HOLs. 

4.8  Ada  PROGRAMMING  LANGUAGE 

As  mentioned  earlier,  after  the  successful 
development  of  the  first  HOLcompiler,  many 
new  HOLs  were  quickly  introduced.  The 
proliferation  of  languages  within  DOD  had 
resulted  in  an  unwieldy  logistics  problem. 
Each  different  language  had  its  own  unique 
support  requirements.  In  1975,  Malcolm 
Currie,  then  Under  Secretary  of  Defense  for 
Research  and  Engineering  (USDRE),  sug¬ 
gested  that  the  DOD  consider  using  just  one 
software  language.  In  1976,  an  interim  policy 
was  issued  requiring  the  use  of  "approved" 
higher  order  languages  listed  in  DOD  Instruc¬ 
tion  5000.3 1.  At  about  the  same  time  a  com¬ 
mittee  called  the  Higher  Order  Language 
Working  Group  was  formed  to  review  existing 
HOLs  to  determine  candidates  for  a  single 
DOD  language.  Their  charter  was  to  look  for 
HOLs  specifically  geared  for  weapon  system 
acquisition.  One  of  their  goals  was  to  find  a 
language  that  supported  both  real-time 
processing  and  large  scale  software  develop¬ 
ment  programs.  The  HOLwas  also  supposed 
to  support  modern  programming  techniques 
and  practices  such  as  top-down  and  structu  red 
design. 

The  Language  Working  Group’s  January 
1977  evaluation  concluded  that  no  existing 
language  met  all  the  DOD  requirements  but 
that  .some,  such  as  Pascal,  Algol,  and  PLl 
could  form  a  good  basis  for  designing  a  "new" 
language.  In  July  1977,  as  a  result  of  an  exten¬ 
sive  evaluation  process,  a  new  set  of  language 
requirements  was  established.  The  govern¬ 
ment  initiated  four  contracts  for  the  design  of 
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this  new  DOD  language.  After  exhaustive 
design  evaluations,  a  final  selection  was  made 
for  one  contractor  to  develop  the  new  lan¬ 
guage  which  became  known  as  Ada.  The  lan¬ 
guage  was  named  after  Lady  Augusta  Ada 
Byron,  Countess  of  Lovelace,  who  is  credited 
with  being  the  first  programmer.  She  was  a 
well  educated  mathematician  who  suggested 
and  wrote  the  first  programs  for  Charles 
Babbage’s  "Analytic  Engine",  a  predecessor  of 
the  modern  computer.  The  winning  contrac¬ 
tor  was  the  French  based  Honeywell-Bull 
Corporation,  and  the  design  team  was  led  by 
Jean  Ichbiah.  Their  product,  the  Ada 
Programming  Language  Specification,  MIL- 
STD-1815,  was  officially  published  on  10 
December  1980.  In  June  1983,  it  became  an 
American  National  Standards  Institute 
(ANSI)  standard  and  was  officially  published 
as  ANSI/MIL-STD-1815A  on  22  January 
1983.  In  March  1987  it  became  an  Interna¬ 
tional  Standards  Organization  (ISO)  stand¬ 
ard.  The  European  software  community  has 
been  quick  to  adopt  Ada. 

In  June  1983  when  Dr.  Richard  DeLauer, 
then  the  USDRE,  directed  that  Ada  be  used 
on  all  new  major  programs,  he  had  been  led 
to  believe  by  the  DOD  software  community 
that  the  necessary  technical  support  (i.e., 
compilers)  for  Ada  would  be  ready.  Unfor¬ 
tunately,  this  was  not  the  case  and  this  lack  of 
supporting  tools  is  probably  responsible  for  a 
majorit;  of  the  bad  initial  publicity  Ada  may 
have  re  :eived.  The  fact  remains,  however, 
that  Ada  is  a  good  language  for  use  on  MCCR 
as  well  as  for  general  purpose  applications. 
What  was  initially  missing  were  the  support 
resources  required  to  develop  Ada  software; 
the  same  resources  that  are  required  to 
develop  software  in  any  HOL.  The  need  for 
support  was  well  recognized  and  initial  •'’orri- 
pilers  and  tool  sets  to  aid  in  the  development 
of  Ada  software  are  now  becoming  readily 
available. 


The  major  concept  behind  the  use  of  Ada  is 
the  enforcement  of  modern  software  en¬ 
gineering  principles.  It  is  the  real  strength 
behind  the  Ada  programming  language. 

4.9  SOFTWARE  ENGINEERING  AND 
TECHNOLOGY 

A  great  deal  of  effort  has  been  applied  to  the 
area  of  software  engineering,  especially  the 
application  of  systems  engineering  to  the 
software  development  process.  Numerous  or¬ 
ganizations  have  been  created  and/or  tasked 
with  evaluating  methods  for  improving 
software  quality  and  reliability;  for  reducing 
development  and  support  costs;  and  for  con¬ 
trolling  the  management  of  software  develop¬ 
ment.  The  main  programs  that  have  been 
initiated  are: 

(a)  Very  High  Speed  Integrated  Circuits 
(VHSIC)  -  The  VHSIC  Program  Office  was 
created  in  1980  by  the  DOD  in  order  to  al¬ 
leviate  deficiencies  in  military  integrated  cir¬ 
cuits  (ICs).  Unlike  the  early  days  of  IC 
technology,  the  DOD  was  no  longer  the  driv¬ 
ing  force  behind  technological  innovation  nor 
the  largest  user  of  IC  chips.  The  commercial 
market  place  was  now  dictating  IC  develop¬ 
ments  by  virtue  of  the  fact  that  it  was  by  far  the 
largest  consumer  of  IC  products.  This  meant 
that  military  applications  had  become  a 
specialty  and  a  highly  customized  business. 
The  commercial  market  place  had  no  need  for 
extremely  fast,  highly  specialized,  low-volume 
IC  products;  therefore  the  DOD  took  the  in¬ 
itiative,  through  the  VHSIC  program,  to  ac¬ 
celerate  the  development  of  this  technology. 

Another  problem  plaguing  military 
electronics  has  been  the  unusually  long  time 
it  takes  to  incorporate  a  new  product  into  a 
weapon  system  once  the  product  is  introduced 
into  the  commercial  market  place.  It  is  not 
unusual  for  a  product  to  appear  in  a  weapon 
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system  up  to  five  years  after  its  commercial 
appearance.  In  order  to  speed  up  this  process, 
the  VHSIC  Program  has  supported  the 
development  and  insertion  of  VHSIC  chips 
into  military  systems.  This  gives  developers 
and  acquisition  managers  a  military  qualified 
microelectronic  technology  that  is  on  a  par 
with  commercially  available  technology.  At 
present,  VHSIC  technology  is  being  intro¬ 
duced  iii'o  at  least  twenty-seven  major  sys¬ 
tems.  This  technology  has  the  potential  for 
greatly  improving  system  performance  [4]. 
Great  advances  are  being  made  in  computer 
hardware.  The  challenge  to  the  software  com¬ 
munity  is  to  capitalize  on  this  wealth  of 
hardware  technology.  Software  needs  to  be 
developed  to  fully  use  the  capability  of  this 
new  hardware. 

(b)  Software  Technology  for  Adaptable  Reli¬ 
able  Systems  (STARS)  -  The  ST  ARS  Pro¬ 
gram  Office  was  established  in  1983  by  the 
DOD  to  investigate  ways  of  reducing  software 
development  costs,  to  increase  software  sys¬ 
tems  reliability,  to  investigate  software  auto¬ 
mation  techniques,  and  to  look  at  applications 
for  reusable  software. 

(c)  Software  Engineering  Institute  (SEI)  - 
The  SEI,  located  at  Carnegie-Mellon  Univer¬ 
sity,  was  placed  under  contract  by  the  Air 
Force  (Electronics  System  Division, 
Hanscom  AFB)  in  1984.  They  were  tasked 
with  investigating  the  transition  of  new 
software  technology,  analyzing  software 
development  environments,  and  providing 
education  in  the  software  and  system  en¬ 
gineering  process.  The  SEI  has  recommended 
changes  to  the  Federal  Acquisition  Regula¬ 
tions  on  software  data  rights  provisions  [5]; 
developed  an  educational  program  on 
software  engineering;  established  liaisons 
with  a  variety  of  educational  institutions  in 
order  to  disseminate  curriculum  information 
and  material  for  undergraduate  and  graduate 


education;  and  conducted  numerous  software 
engineering  conferences. 

(d)  Defense  Science  Board  (DSB)  -  A  DSB 
Task  Force  on  software  was  originally  con¬ 
vened  in  1981  by  the  Under  Secretary  of 
Defense  (Acquisition)  to  review  a  draft  DOD 
Instruction  on  standardizing  computer 
hardware.  The  DSB  recommended  cancelling 
any  further  tasking  in  this  area.  The  Air  Force, 
however,  had  alrerjy  developed  a  stand¬ 
ardized  ISA  for  a  sixteen  bit  airborne  com¬ 
puter  (MIL-STD-1750A).  Later  in  the  year, 
the  DSB  was  tasked  with  reviewing  overall 
software  acquisition,  management,  and  com¬ 
puter  resource  technology  procedures  and 
providing  recommendations  for  rectifying  any 
problems.  The  Task  Force’s  September  1987 
report,  stated  that  the  major  problems  with 
military  software  development  were  not  tech¬ 
nical  problems,  but  management  problems. 
They  recommended  that  the  DOD  re-ex¬ 
amine  and  change  the  attitudes,  policies,  and 
practices  regarding  software  acquisition  [5]. 

4.10  SOFTWARE  SUPPORT 

The  Joint  Logistics  Commanders  Joint  Policy 
Coordination  Group  on  Computer  Resources 
Management  established  a  sub-panel  in  1979 
to  specifically  look  into  post-deployment 
software  support  (PDSS)  and  the  procedures 
required  to  provide  adequate  software  sup¬ 
port  during  and  after  transition.  PDSS  will  be 
di.scussed  in  Chapter  7. 

4.11  TOP  LEVEL  SERVICE  DIRECTIVES 
AND  GUIDELINES 

The  following  is  a  list  of  the  guidance  docu¬ 
ments  for  the  respective  services.  All  of  these 
documents  stem  from  the  guidance  provided 
by  DOD  Directive  5000.29,  Management  of 
Computer  Resources  in  Major  Defense  Sys¬ 
tems. 
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Navy 

SECNAVINST  520032 

Management  of  Embedded  Computer  Resour¬ 
ces  in  Department  of  the  Navy  Systems,  Jun  79 

OPNAVINST  5200.28 

Life  Cycle  Management  of  Mission  Critical 
Computer  Resources  for  Navy  Systems,  25  Sep¬ 
tember  1985 

OPNAVINST  5230.21 

Instruction  on  Standard  Embedded  Computer 
Resources 

NAVELEXINST  5200.23 

Instruction  on  General  Software  Management 

TADSTANDS 

Tactical  Digital  Systems  Standards  A  thwugh 
D  on  standard  definitions,  computers, 
programming  languages,  reserve  capacities, 
and  documentation. 

Air  Force 

APR  800-14 

Life  Cycle  Management  of  Computer  Resour¬ 
ces  in  Systems,  29  September  1986 

AFSCP  800-14 

Air  Force  Systems  Command  Software  Quality 
Indicators:  Management  Quality  Insight,  20 
January  1987 

AFSCP  800-43 

Air  Force  Systems  Command  Software  Quality 
Indicators:  Management  Insight,  31  Jan  1986 

AFSC/AFLCP  800-45 

Software  Risk  Abatement  (Draft),  1988. 

AFSCP  800-5 

Software  Independent  Verification  and  Valida¬ 
tion  (IV&V)  (Draft) 


ASDP  800-5 

Software  Development  Capability/Capacity 
Review,  10  1987 

Army 

DARCOM-R.70-16 

Management  of  Computer  Resources  in  Bat¬ 
tlefield  Automated  Systems,  16  July  1979 

Assistant  Secretary  of  Army  Policy  Letter 

Standardization  of  ECR,  1  July  1980 

AMC-P  70-13 

AMC  Software  Management  Indicators,  31 
January  1987 

Marines 

MCO  5200.23 

Management  of  ECR  in  the  Marine  Corps,  19 
August  1982 

4.12  SOFTWARE  DATA  RIGHTS 

The  following  definitions  are  relevant  to 
software  data  rights: 

Copyright  -  A  copyright  protects  the  expres¬ 
sion  of  an  idea  through  unauthorized  copying 
or  reproduction.  It  is  easy  and  inexpensive  to 
obtain  by  simply  filing  an  application  with  the 
copyright  office  and  providing  a  copy  for  the 
Library  of  Congress.  No  examination  of  the 
material  is  required. 

Trade  secret  -  Protects  the  underlying  ideas, 
concepts,  procedures,  formula,  pattern, 
device  nr  compilation  that  derives  economic 
value  by  not  being  readily  available  to  others. 
Must  maintain  secrecy  (information  is  not 
public  domain). 

Patent  -  Protection  by  trademark  or  trade 
name.  It  is  expensive,  uncertain,  and  time 
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consuming  (may  take  two  years  to  obtain). 
Patents  destroy  trade  secret  protection  be¬ 
cause  patent  disclosures  are  quite  complete. 

Restricted  rights  -  Software  developed  totally 
by  non-government  funds  and  usually 
licensed  to  users. 

License  agreements  -  Agreement  between  a 
contractor  and  the  government  (or  another 
contractor)  limiting  the  use  and  the  copying 
of  data  which  has  been  commercially  sold 
(rights  to  use,  disclose,  or  reproduce). 

The  Federal  copyright  laws,  patent  laws,  and 
state  trade  secret  laws  provide  legal  protec¬ 
tion  of  a  contractor’s  computer  software.  The 
Copyright  Act  of  1976  was  amended  in  1980 
to  include  computer  programs.  The  Office  of 
Federal  Procurement  Policy  (OFPP)  was 
tasked  by  a  10  April  1987  Executive  Order,  to 
develop  a  firm  national  policy  in  favor  of  com¬ 
mercial  rights  [6].  In  the  meantime,  FAR  Sub¬ 
part  27.4  and  52.227-14  provide  guidance  to 
the  program  office  on  software  data  rights. 
Specific  language  in  accordance  with  these 
clauses  of  the  FAR  should  be  included  in  the 
contract,  both  for  the  protection  of  the 
government  and  the  contractor. 

4,13  AUTOMATED  INFORMATION  SYS¬ 
TEMS 

Although  this  chapter  deals  primarily  with  the 
computer  resources  associated  with  weapon 
systems,  it  may  be  useful  to  briefly  discuss  the 
DOD  policy  on  automated  information  sys¬ 
tems. 

After  the  DOD  had  taken  steps  to  provide 
initial  guidance  on  the  management  aspects 
of  weapon  system  computer  resources,  it  pub¬ 
lished  DOD  Directive  7920.1,  Life  Cycle 
Management  of  Automated  Information  Sys¬ 
tems  (AIS)  on  17  October  1978  and  DOD 


Instruction  7920.2,  "Major  Automated  Infor¬ 
mation  Systems  Approval  Process"  on  20  Oc¬ 
tober  1978.  These  two  documents  established 
the  policy  for  the  procurement  of  general  pur- 
po.se  computers  or  ADP  equipment  primarily 
intended  for  u.se  in  business  applications  and 
subject  to  the  provisions  of  the  Brooks  Bill. 
AIS  defines  procedures  for  the  acquisition  of 
equipment  which  is  designed,  built,  operated, 
and  maintained  for  the  sole  purpose  of  col¬ 
lecting,  recording,  processing,  storing, 
retrieving,  and  displaying  information.  These 
systems  usually  have  large  data  storage  re¬ 
quirements  and  are  used  for  business  type 
applications  such  as  payroll,  accounting,  and 
inventory.  The  function  of  these  two  DOD 
documents  is  very  similar  to  that  of  the  DOD 
Directive  5000.29.  Some  of  the  similarities 
include  the  promotion  of  life  cycle  manage¬ 
ment,  visibility,  cost  effectiveness,  stand¬ 
ardization  of  the  approval  process,  and 
emphasis  on  requirements’  validation. 

4.14  SUMMARY 

Proliferation  of  software  and  computer 
resources  has  occurred  since  their  introduc¬ 
tion  in  the  late  fifties.  The  Brooks  Bill  was 
pas.sed  in  1965  to  promote  competition  and  to 
regulate  how  ADP  should  be  acquired  and 
managed  within  the  government.  The  em¬ 
phasis  in  this  chapter,  however,  has  been  on 
the  acquisition  policy  for  software  exempted 
from  the  Brooks  Bill. 

In  1976  the  DOD  issued  Directive  5()()().29. 
This  directive  provided  guidance  on  the 
management  of  software  and  embedded  com¬ 
puter  resources.  Management  Steering  Com¬ 
mittee  for  Embedded  Computer  Resources 
was  established  to  guide  this  effort.  This  was 
followed  by  software  workshops  in  1979  con¬ 
ducted  by  the  Joint  Lttgistics  Commanders’ 
Joint  Policy  Coordination  Group  on  Com¬ 
puter  Resource  Management.  For  the  past 
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several  years,  a  major  DOD  focus  has  been 
the  standardization  of  the  software  life  cycle. 
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CHAPTERS 


SOFTWARE  DEVELOPMENT  PROCESS 


5.1  INTRODUCTION 

The  development  of  a  weapon  system  re¬ 
quires  integrating  technical,  administrative, 
and  management  disciplines  into  a  cohesive, 
well-planned,  and  rigorously  controlled 
process.  As  a  critical  component  of  a  weapon 
system,  software  must  be  developed  under  a 
similarly  disciplined  engineering  process.  In 
Software  Engineering  Concepts  [1],  Richard 
Fairley  defines  software  engineering  as: 

"...the  technological  and  managerial  discipline 
concerned  with  systematic  production  and 
maintenance  of  software  products  that  are 
developed  and  modified  on  time  and  within  cost 
estimates." 

Barry  Boehm  [2]  defines  software  engineer¬ 
ing  as  a  discipline  that: 

"...involves  the  practical  application  of  scientific 
knowledge  to  the  design  and  construction  of 
computer  programs  and  the  associated  docu¬ 
mentation  required  to  develop,  operate,  and 
maintain  them." 


The  main  point  is  that  the  software  develop¬ 
ment  process  must  be  scientific  and  dis¬ 
ciplined.  This  is  not  different  from  the 
hardware  development  process.  As  with 
hardware,  the  goal  of  the  soft. /are  develop¬ 
ment  process  is  to  consistently  produce  a 
quality  product,  within  schedule  and  cost. 

With  the  publication  of  DOD-STD-2167A, 
Defense  System  Software  Development,  the 
DOD  took  the  first  step  toward  a  stand¬ 
ardized,  systems  engineering  approach  to 
software  development  [3].  This  standard  is 
supported  by  other  military  documents  and 
describes  a  standard  process  and  documenta¬ 
tion  for  computer  software  development.  To 
use  this  standard  effectively,  the  program  of¬ 
fice  must  have  a  thorough  understanding  of 
the  system  being  developed;  particularly  the 
overall  system  requirements  and  con¬ 
straints. 
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requirements  must  be  maintained  throughout 
the  acquisition  life  cycle  and  any  requirement 
that  cannot  be  traced  up  to  a  higher  require¬ 
ment  should  be  modified  or  eliminated. 

The  material  presented  in  this  chapter  will 
describe  activities  that  occur  in  a  "typical"  pro¬ 
gram.  The  reader  should  understand  that  real 
programs  seldom  actually  follow  this  "typical" 
profile.  Phases  can  occur  concurrently;  they 
can  be  by-passed  altogether;  protracted;  or 
condensed  to  satisfy  the  needs  of  the  overall 
program  objectives.  The  point  to  understand 
is  that  although  the  process  is  somewhat  con¬ 
stant,  its  chronological  occurrence  is  not 
fixed.  The  following  sections  describe  the 
classical  approach  to  software  development. 

5.2  SUMMARY  OF  DEVELOPMENT  AC¬ 
TIVITIES 

Figure  5-1  presents  an  overview  of  the 
development  activities  of  an  integrated 
software  and  hardware  system  as  reflected  in 
DOD-STD-2167A. 


HWRqmIs 

Analysis 


All  weapon  system  development  programs 
begin  with  a  determination  of  system  level 
re'  iirements.  These  activities  occur  during 
the  Concept  Exploration  (CE)  and  the 
Demonstration/Validation  (D/V)  phases  of 
the  acquisition  cycle. 

The  Systems  Requirements  Review  (SRR) 
may  be  held  after  the  initial  determination  of 
system  functions  (functional  analysis)  and  the 
preliminary  allocation  of  these  functions  to 
configuration  items.  The  SRR  provides  an 
opportunity  for  an  initial  insight  into  the 
developer’s  direction,  progress  and  conver¬ 
gence  on  a  system  configuration.  The  System 
Design  Review  (SDR)  is  a  review  of  the  over¬ 
all  system  requirements  in  order  to  establish 
the  functional  baseline  documented  by  the 
system  specification.  The  functional  baseline 
should  allocate  requirements  to  hardware  and 
software  configuration  items. 

The  development  of  both  hardware  and 
software  can  begin  once  the  Functional 
Baseline  is  established.  These  activities  occur 
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in  the  Full  Scale  Development  (FSD)  phase 
and  are  monitored  through  reviews  and  audits 
as  described  in  MIL-STD-1521B,  Technical 
Reviews  and  Audits  for  Systems,  Equipment, 
and  Computer  Resources.  The  Allocated 
Baseline  for  software  should  be  established  at 
the  Software  Specification  Review  (SSR).  For 
hardware  the  allocated  baseline  is  normally 
established  at  the  Preliminary  Design  Review 
(PDR),  or  no  later  than  the  Critical  Design 
Review  (CDR). 

Product  development  starts  once  the  design 
effort  is  completed.  (For  systems  using  new 
functions,  procedures,  and  techniques,  it  may 
be  necessary  to  actually  perform  some  trial 
coding  and  testing  in  order  to  complete  the 
design.)  For  hardware  this  building  effort  is 
called  fabrication  and  for  software  it  is  called 
coding  and  testing.  Testing  is  further  sub¬ 
divided  into  Computer  Software  Unit  (CSU) 
testing  and  Computer  Software  Component 
(CSC)  integration  and  testing.  After  the 
items  are  built,  formalized  testing  takes  place 
in  accordance  with  approved  test  plans  and 
procedures.  A  government  Test  Readine.ss 
Review  (TRR)  is  conducted  to  determine  the 
developer’s  readiness  to  perform  formalized 
acceptance  testing.  Completion  of  software 
testing  will  lead  to  .system  integration  and 


testing.  Both  F'unctional  Configuration 
Audits  (FCA)  and  Physical  Configuration 
Audits  (PCA)  will  be  conducted  on  hardware 
and  software  configuration  items  to  establish 
the  respective  Product  Baselines.  After  a  sys¬ 
tem  level  Formal  Qualification  Review 
(FOR),  the  integrated  .system  is  turned  over 
to  the  government  for  operational  testing  as 
defined  in  the  system’s  Test  and  Evaluation 
Master  Plan  (TEMP).  Succe.ssful  completion 
of  this  testing  indicates  that  the  product  is 
fully  defined  and  ready  to  be  manufactured. 
For  hardware,  the  production  line  would 
begin  to  assemble  carbon  copy  items.  For 
software,  turning  out  copies  is  a  trivial 
process.  The  product  is  complete  and  needs 
only  to  be  duplicated  on  the  required  media 
for  transfer  to  the  target  .system  computer. 

5.3  SYSTEM  REQUIREMENTS  ANALYSIS 
AND  DESIGN 

Figure  5-2  depicts  the  activities  and  products 
associated  with  the  CE  and  D/V  phases.  The 
CE  and  D/V  phase  activities  are  .system 
oriented  to; 

(a)  Define  overall  project  objectives; 

(b)  Determine  project  feasibility; 
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(c)  Develop  acquisition  development 
strategy; 

(d)  Establish  resource  cost  and  schedule; 

(e)  Define  the  interrelationships  between 
hardware  and  software; 

(f)  Define  technical  and  business  functions 
and  performance. 

The  first  step  is  to  generate  the  system  level 
requirements  and  reflect  them  in  a  System/ 
Segment  Specification  (SSS)  (Type  A 
Specification).  It  doesn’t  make  any  difference 
whether  it  is  a  hardware  only,  a  software  only, 
or  a  hardware  and  software  system;  the  most 
important  and  critical  aspect  of  weapon  sys¬ 
tem  development  is  to  "nail  down  '  the  system 
requirements  which  must  first  be  finalized  at 
the  functional  level,  before  being  allocated  to 
hardware  and  software.  Recognize  that  newer 
software  development  practices  use  trial 
coding  and  testing  as  a  method  of  refining 
derived  software  requirements. 

The  requirements,  finalized  through  a  series 
of  engineering  studies  and  tradeoffs,  include: 

(a)  Requirements  Refinement  -  The  overall 
system  requirements,  including  constraints, 
should  be  examined  to  identify  the  factors  that 
drive  requirements  for  computer  resources. 
These  factors  may  include  system  interfaces, 
interoperability,  communication  functions, 
personnel  functions,  the  anticipated  level  and 
urgency  of  change,  and  requirements  for 
reliability  and  responsive  support. 

(b)  Operational  Concept  Analysis  -  The 
operational  concept  should  be  analyzed  in 
order  to  determine  the  role  of  computer 
resources.  Particular  attention  is  paid  to  re¬ 
quirements  for  mission  preparation,  operator 


interface,  control  functions,  and  mission 
analysis. 

(c)  Tradeoff  and  Optimization  -  The  effects 
of  system  constraints  such  as  the  operations 
concept,  the  support  concept,  performance 
requirements,  logistics,  availability  and 
maturity  of  technology,  and  limitations  on 
cost,  schedule,  and  resources  are  determined. 
Alternative  computer  resources  approaches 
are  studied  to: 

-  meet  operational,  interoperability,  and 
support  requirements; 

-  determine  how  the  system  requirements 
for  reliability  and  maintainability  will  be 
satisfied; 

-  determine  how  requirements  for  system 
security  will  be  met; 

A  determination  will  al.so  be  made  regarding 
the  suitability  of  standard  computer  lan¬ 
guages  and  instruction  set  architectures. 

(d)  Risk  -  For  each  approach,  the  risk  as¬ 
sociated  with  computer  resources  is 
evaluated.  Risk  areas  include  compiler 
maturity,  availability  and  maturity  of  the 
software  support  tools,  loosely  defined  or  in¬ 
complete  interface  definitions,  and  lack  of 
adequate  computer  memory  or  throughput 
capability. 

53.1  System  Design 

System  Design  begins  m  or  about  the  time  of 
the  SRR.  The  major  function  of  System 
Design  is  to  establish  the  functional  baseline 
of  the  system  by  updating  and  approving  the 
system  specification  and  the  operational  con¬ 
cept;  by  developing  the  initial  subsystem/seg¬ 
ment  designs;  and  by  refining  the  systems 
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engineering  planning  activities  to  be 
employed  during  system’s  development. 
Typical  products  are: 

(a)  System  Specification; 

(b)  System/Segment  Designs; 

(c)  Configuration  Management  Plan  (CMP); 

(d)  Computer  Resources  Life  Cycle  Manage¬ 
ment  Plan  (CRLCMP); 

(e)  Preliminary  Software  Requirement 
Specification  (SRS); 

(f)  Preliminary  Interface  Requirements 
Specification  (IRS); 

5.4  SOFTWARE  DEVELOPMENT 

Before  discussing  software  development, 
some  definitions  are  in  order: 

Computer  Software  Configuration  Item 
(CSCI)  -  A  configuration  item  for  computer 
software. 

Computer  Software  Component  (CSC)  -  A 
distinct  part  of  a  computer  software  con¬ 
figuration  item  (CSCI).  CSCs  may  be  further 
decomposed  into  other  CSCs  and  Computer 
Software  Units. 

Computer  Software  Unit  -  An  element 
specified  in  the  design  of  a  Computer 
Software  Component  (CSC)  that  is  separately 
testable.  A  CSU  is  the  lowest  level  of  software 
decomposition. 

Weapon  system  software  is  partitioned  into 
CSCIs  based  on  the  program  office’s  manage¬ 
ment  strategy.  Each  CSCI  is  managed  in¬ 
dividually  and  follows  its  own  development 
process.  A  software  development  proce.ss  is 


defined  in  DOD-STD-2167A  and  consists  of 
eight  major  activities:  Systems  Requirements 
Analysis/Design,  Software  Requirements 
Analysis,  Preliminary  Design,  Detailed 
Design,  Coding  and  CSU  Testing,  CSC  In¬ 
tegration  and  Testing,  and  CSCI  Testing, 
Systems  Integration  and  Testing.  These 
steps  typically  occur  during  FSD,  although 
they  may  occur  one  or  more  times  during  each 
of  the  system  life  cycle  phases  [4].  This  is 
especially  true  if  software  prototyping  is  per¬ 
formed  during  the  Concept  Demonstration 
and  Validation  Phase.  The  steps  are  not 
linear  since  software  development  is  iterative 
in  nature  and  any  step  may  be  repeated  many 
times  during  the  course  of  system  develop¬ 
ment.  For  many  software  developments  it  is 
necessary  to  plan  for  these  iterations  prior  to 
establishing  a  firm  allocated  baseline. 

Managing  software  is  very  similar  to  manag¬ 
ing  hardware;  both  require  discipline  and 
control  in  order  to  succeed.  An  important  part 
of  the  control  process  is  the  formal  deter¬ 
mination  of  whether  or  not  the  developer  is 
ready  to  proceed  to  the  next  step.  This  is 
usually  determined  through  a  series  of  design 
reviews  and  audits.  Software  reviews  and 
audits  can  occur  in  conjunction  with  hardware 
reviews,  but  they  do  not  necessarily  have  to. 
It  is  important  that  appropriate  system  level 
reviews  be  held  at  strategic  intervals.  This  will 
focus  everyone’s  attention  on  system  design 
and  leads  to  timely  baselines  for  the  hardware, 
the  software,  and  all  the  interfaces.  Software 
development  has  two  major  reviews  that  are 
separate  from  hardware  reviews:  the  Software 
Specification  Review  (SSR)  and  the  Test 
Readiness  Review  (TRR). 

The  SSR  is  a  formal  review  of  a  CSCI’s  re¬ 
quirements  as  specified  in  the  software 
specifications.  A  collective  SSR  for  a  group  of 
configuration  items,  addressing  each  con¬ 
figuration  item  individually,  may  be  held 
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when  such  an  approach  is  advantageous  to  the 
government.  Its  purpose  is  to  establish  the 
allocated  baseline  for  preliminary  CSCI 
design  by  demonstrating  to  the  government 
the  adequacy  of  the  software  specifications. 

The  TRR  is  a  formal  review  of  the 
contractor’s  readiness  to  begin  formal  CSCI 
testing.  It  is  conducted  after  software  test 
procedures  are  available  and  CSC  Integration 
testing  is  complete.  The  purpose  of  the  TRR 


based  on  the  System  Specification.  The 
means  of  testing  and  examining  the  software 
are  also  identified.  During  requirements 
analysis,  prototype  versions  of  high  risk  areas, 
user  interfaces,  and/or  systems  skeletons  may 
be  partially  designed  and  coded.  Prototyping 
is  an  excellent  tool  for  performing  require¬ 
ments  analysis. 

The  developer  should  also  identify  support 
tools  and  resources,  and  establish  timing  and 
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is  to  determine  whether  the  contractor  is 
ready  to  begin  formal  CSCI  testing  that  can  be 
witnessed  by  the  government.  A  technical 
understanding  must  be  reached  on  the  infor¬ 
mal  test  results,  and  on  the  validity  and  the 
degree  of  completeness  of  such  documents  as 
an  operator’s  manual,  a  user’s  manual,  and  a 
computer  programmer’s  manual. 

5.4.1  Software  Requirements  Analysis 

The  first  step  in  the  software  development 
cycle  is  the  Software  Requirements  Analysis 
(Figure  5-3).  The  purpose  of  the  Software 
Requirements  Analysis  is  to  establish 
detailed  functional,  performance,  interface, 
and  qualification  requirements  for  each  CSCI 


sizing  estimates.  The  Program  Manager  must 
ensure  that  all  software  requirements,  as 
reflected  in  the  software  development 
specifications,  are  traceable  to  the  system 
specification  and  that  the  Software  Develop¬ 
ment  Plan  is  updated  to  identify  the  required 
resources,  facilities,  personnel,  development 
schedule  and  milestones,  and  software  tools. 
The  developer  may  also  customize  the  techni¬ 
ques,  methodologies,  standards  and  proce¬ 
dures  to  be  used  in  software  development. 

The  outputs  of  the  Software  Requirements 
Analysis  are  final  versions  of  the  software 
specifications,  and  an  updated  Software 
Development  Plan.  These  documents  will  be 
reviewed  at  the  SSR.  The  Computer  Resour- 
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ces  Life  Cycle  Management  Plan  (CRLCMP) 
may  also  be  updated. 

5.4.2  Preliminary  Design 

After  the  software  allocated  baseline  is  estab¬ 
lished,  the  developer  has  traditionally 
proceeded  into  the  Software  Preliminary 
Design  as  shown  in  Figure  5-4.  Preliminary 
design  activity  determines  the  overall  struc¬ 
ture  of  the  software  to  be  built.  Based  on  the 
requirements,  the  developer  may  partition 
the  software  into  components  and  define  the 
function  of  each  component  and  the  relation¬ 
ships  between  them.  This  is  called  functional 
decomposition.  Input  and  output  relation¬ 
ships  with  external  devices  (such  as  displays 
and  sensors)  are  refined  according  to  the 
hardware  configuration  and  software  struc¬ 
ture.  Timing  and  memory  budgets  for  com¬ 
ponents  are  established  so  that  the  software 
requirements  can  be  satisfied  within  the 
hardware  constraints.  If  a  technique  such  as 
Object  Oriented  Design  (OOD)  is  used,  func¬ 
tional  decomposition  doesn’t  really  apply.  In¬ 
stead  objects  and  classes  are  established  along 
with  clear  definition  of  data  requirements. 
The  developer  should  provide  a  preliminary 
design  that  insures  that  requirements  from 


the  software  specifications  can  be  traced 
down  to  the  software  components  of  each 
CSCI.  In  effect  the  early  work  on  preliminary 
design  can  be  used  to  validate  the  allocated 
baseline.  The  software  design  is  reflected  in 
the  preliminary  Software  Design  Document 
(SDD)  and  Interface  Design  Document 
(IDD).  These  documents  will  describe  the 
system  architecture,  memory  and  processing 
time  allocations,  interrupt  requirements, 
timing  and  sequencing  considerations,  and 
input/output  constraints  for  each  software 
component.  The  developer  should  also 
generate  a  Software  Test  Plan  (STP)  outlining 
the  proposed  test  program  and  establishing 
test  requirements  for  software  integration 
and  testing. 

The  outputs  of  the  contractor’s  efforts  are 
preliminary  versions  of  the  software  design 
documents  and  the  Software  Test  Plan.  These 
documents  are  reviewed  during  the  PDR. 
Throughout  the  development  effort,  the 
developer  will  conduct  informal  design 
reviews,  inspections,  and  walkthroughs  to 
evaluate  the  progress  and  correctness  of  the 
design  for  each  software  component.  The 
results  of  these  inspections  will  serve  as  the 
basis  for  material  presented  at  the  PDR. 
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5.4.3  Detailed  Design 

The  purpose  of  the  Detailed  Design  (Figure 
5-5)  activity  is  to  logically  define  anu  complete 
a  software  design  that  satisfies  the  allocated 
requirements.  The  level  of  design  detail  must 
be  such  that  development  of  the  computer 
program  can  be  accomplished  by  someone 
other  than  the  original  designer.  Component 
functions,  inputs  and  outputs,  plus  any  con¬ 
straints  (such  as  memory  size  or  response 
time)  should  be  defined.  Logical,  static,  and 
dynamic  relationships  among  the  components 
should  be  specified  and  the  component  and 
system  integration  test  procedures  generated. 

A  complete  detailed  design  includes  not  only 
a  description  of  the  computer  processes  to  be 
performed  but  also  detailed  descriptions  of 
the  data  to  be  processed.  A  data  dictionary  is 
an  effective  way  of  documenting  this  needed 
design  information.  For  software  that  proces¬ 
ses  or  manipulates  a  large  amount  of  interre¬ 
lated  data,  the  structure  of  the  data  itself 
should  be  defined. 


guage  should  be  clearly  defined  and  the 
reasons  for  the  departure  from  the  standard 
justified.  Any  special  conditions  that  must  be 
followed  when  programming  the  component 
should  be  similarly  described  and  clearly 
documented  [5].  These  exceptions  are  nor¬ 
mally  addressed  in  the  Software  Develop¬ 
ment  Plan. 

During  the  entire  design  and  development 
process  the  contractor  should  document  the 
development  of  each  unit,  component,  and 
CSCI  in  software  development  folders 
(SDFs).  A  separate  SDF  should  be  main¬ 
tained  for  each  division  or  breakdown  of  the 
software.  These  divisions  or  breakdowns  are 
determined  by  the  particular  design 
methodology  used.  The  SDFs  are  normally 
maintained  for  the  duration  of  the  contract 
and  made  available  for  government  review 
upon  request.  A  set  of  SDFs  may  include  the 
following  information: 

(a)  Design  considerations  and  constraints; 


(b)  Design  documentation  and  data; 

Components  which  must  be  coded  in  assemb¬ 
ly  language  or  another  "non-standard  ’  Ian-  (c)  Schedule  anu  status  information; 
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(d)  Test  requirements  and  responsibilities; 

(e)  Test  cases,  procedures  and  results. 

The  contractor  documents  and  implements 
procedures  for  establishing  and  maintaining 
SDFs  in  a  Software  Development  Library. 
The  library  is  a  management  tool  used  by  the 
contractor  io  assist  in  developmental  con¬ 
figuration  management.  It  serves  as  a  "storage 
house"  to  control  access  of  software, 
documentation,  and  associated  tools  and  pro- 


Fig.  5-6  Coding  and  CSU  Testing 

cedures  used  to  facilitate  the  orderly  develop-  If  the  detailed  design  is  in  error,  is  ambiguous, 

ment  and  subsequent  support  of  software  [6].  or  is  not  sufficiently  complete  to  permit  the 

programming  to  continue  without  further 
A  CDR  (or  a  series  of  smaller  CDRs)  is  con-  definition,  the  programmer  should  consult 

ducted  at  the  conclusion  of  the  detailed  the  original  designer.  The  resolution  should 

design.  The  CDR  should  assure  that  the  be  documented,  and  all  affected  require- 

software  design  satisfies  the  requirements  of  ments,  design,  and  test  documentation  up- 

both  the  system  level  specification  and  the  dated  accordingly, 

software  development  specifications. 

The  purpose  of  the  unit  testing  activity  is  to 
5.4.4  Coding  and  CSU  Testing  eliminate  any  errors  that  may  exist  in  the  units 

as  they  are  programmed.  These  errors  may  be 
The  purpose  of  programming  is  to  translaiv-  due  to  programmer  mistakes  or  deficiencies 

the  detailed  software  design  into  a  program-  in  the  software  requirements  and  design 

ming  language  such  as  Ada.  It  is  during  the  documentation.  Usually,  the  test  of  a  unit  is 

programming  activity  that  listings  of  the  the  responsibility  of  the  programmer  who 

source  program  are  generated  (Figure  5-6).  programmed  the  unit.  Unit  testing  is  the  ac- 

Based  on  the  detailed  software  design  tivity  that  permits  the  most  control  over  test 


presented  in  the  design  specification, 
programming  of  each  unit  is  accomplished  by 
the  assigned  programmer  in  the  specified 
programming  language,  usually  Ada.  As  the 
programming  of  each  unit  is  completed,  the 
programmer  examines  the  program  for  er¬ 
rors.  The  program  may  be  compiled  when  the 
programmer  is  satisfied  that  the  source  pro¬ 
gram  correctly  implements  the  detailed 
design.  Compiling  translates  the  source  pro¬ 
gram  to  its  machine  executable  form,  the  ob¬ 
ject  program. 
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conditions  and  visibility  into  software  be¬ 
havior.  An  efficient  software  development 
effort  requires  rigorous  unit  level  test  to 
detect  most  errors  before  CSC  Integration 
and  Test. 

Besides  producing  the  source  and  object  code 
and  their  listings,  the  contractor  develops  and 
records  in  software  development  folders  the 
informal  test  procedures  for  each  unit  test  as 
well  as  the  test  results.  The  contractor  will 
usually  conduct  informal  code  inspections  or 
walkthroughs  on  each  coded  unit  and  com¬ 
ponent  during  several  stages  of  its  develop¬ 
ment.  There  are  no  formal  reviews  scheduled 
during  this  step  of  the  development  cycle. 

5.4.5  CSC  Integration  and  Testing 

Once  the  software  is  coded  and  each  unit  and 
component  tested  for  compliance  with  its 
design  requirements,  the  contractor  should 
begin  CSC  Integration  and  Testing  as  il¬ 
lustrated  in  Figure  5-7.  The  purpose  of  CSC 
Integration  and  Testing  is  to  combine  the 
software  units  and  components  that  have 
been  independently  tested  into  the  total 
software  product  and  to  demonstrate  that  tlr*' 
combination  fulfills  the  system  design.  The 


integration  is  done  in  a  phased  manner  with 
only  a  few  components  being  combined  at 
first,  additional  ones  added  after  the  initial 
combination  has  been  tested,  and  the  process 
repeated  until  all  components  have  been  in¬ 
tegrated.  The  phasing  of  this  integration 
should  be  based  on  the  functional  capabilities 
that  can  be  demonstrated  by  specific  groups. 
There  may  be  some  overlap  with  the  previous 
step  since  some  components  may  be  ready  for 
integration  while  others  are  still  being  coded. 
Most  testing  performed  during  Coding  and 
eSU  Testing,  and  CSC  Integration  and  Test¬ 
ing  is  called  "informal  testing".  This  term 
doesn’t  imply  that  the  testing  is  "casual"  or 
"haphazard",  but  instead  implies  that  the  test¬ 
ing  doesn’t  require  government  approval. 
Some  formal  testing  may  he  accomplished 
during  these  steps,  but  most  formal  testing  is 
usually  accomplished  during  the  next  step. 

5.4.6  CSCI  Testing 

After  completion  of  a  successful  TRR,  the 
contractor  will  proceed  with  CSCI  Testing 
(Figure  5-8),  the  last  step  of  the  software 
development  cycle.  The  purpose  is  to  per¬ 
form  formal  tests,  in  accordance  with  the 
software  test  plans  and  procedures,  on  each 
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CSCI  and  to  establish  the  software  Product 
Baseline.  Testing  during  this  step  is  intended 
to  show  that  the  software  satisfies  the 
Software  Requirements  Specification  and  the 
Interface  Requirements  Specification. 


results  and  reviewing  the  operational  and  sup¬ 
port  documentation.  The  PCA  is  the  formal 
technical  examination  of  the  as-built  software 
product  against  its  design.  This  includes  the 
product  specification  and  the  as-coded 
documentation. 


Throughout  CSCI  testing,  the  contractor 
should  be  updating  all  previous  software 
documentation,  analyzing  test  data,  generat¬ 
ing  the  Software  Test  Reports  (STR),  and 
finalizing  the  Software  Product  Specification 
(SPS)  (C-5  Specification).  This  will  be  the 
basis  for  the  software  Product  Baseline  nor¬ 
mally  established  at  the  PCA,  which  may  im- 
mediately  follow,  or  be  conducted 
concurrently  with,  the  FCA  for  a  software 
only  development.  Normally,  the  PCA  occurs 
after  the  software  is  released  for  integration 
and  testing  with  the  system  following  the 
software  FCA  as  illustrated  in  Figure  5-8, 
During  the  software  FCA  the  government 
verifies  that  the  CSCIs  perform  in  accordance 
with  their  respective  requirements  and  inter¬ 
face  specifications  by  examining  the  test 


The  typical  outputs  of  the  contractor’s  efforts 
in  CSCI  Testing  are  the  Software  Test  Report 
(STR),  operational  and  support  documenta¬ 
tion  such  as  the  Computer  System  Operator’s 
Manual  (CSOM),  the  Software  Users  Manual 
(SUM),  the  Software  Programmer’s  Manual 
(SPM),  the  Firmware  Support  Manual 
(FSM),  the  Computer  Resources  Integrated 
Support  Document  (CRISD),  the  Version 
Description  Document  (VDD),  and  the 
Software  Product  Specification  (SPS).  Except 
for  updates  and/or  revisions,  all  deliverable 
documentation  should  be  completed  at  this 
time.  Appendix  H  contains  a  listing  of  the 
standardized  software  documentation, 
defined  in  DOD-STD-2167A,  that  may  be 
required  for  software  development  programs. 
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5.5  SYSTEM  INTEGRATION  &  TESTING 

The  purpose  of  System  Integration  and  Test¬ 
ing  is  to  ensure  that  the  developed  software 
works  with  the  system  in  the  environment  that 
it  was  designed  for  (Figure  5-9).  The  system 
is  turned  over  to  the  government  after  an 
acceptable  Formal  Qualification  Review 
(FOR).  The  FOR  is  a  system-level  review  that 
verifies  that  the  actual  system  performance 
complies  with  the  system  requirements.  For 
computer  resources,  it  addresses  the  aspects 
of  the  software  and  hardware  performance 
that  have  been  tested  after  the  FCA  and  PCA. 


stipulated  in  the  contract  will  be  delivered  to 
the  government  who  will  then  assume  con¬ 
figuration  control  responsibility.  The  contrac¬ 
tor,  however,  will  be  available  to  support  the 
government’s  test  and  evaluation  efforts  and 
to  conduct  any  required  acceptance  tests. 

5.6  TAILORING 

The  purpose  of  tailoring  is  to  reduce  the  over¬ 
all  costs  of  an  acquisition,  primarily  by  reduc¬ 
ing  the  amount  and  type  of  documentation 
being  delivered  by  the  contractor  and  by 
eliminating  redundant  or  unnecessary  testing 
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Fig.  5-9  Systems  Integration  and  lest 


A  successful  I'QR  is  predicated  on  a  deter¬ 
mination  that  the  .system  meets  the  s[)ccificd 
requirements  in  the  hardware,  software  and 
interface  specifications. 

The  contractor’s  role  will  diminish  sig¬ 
nificantly  subsequent  to  the  I-'OR.  Contractor 
configuration  control  of  the  software  should 
terminate  once  the  product  baseline  is  ap¬ 
proved  and  the  government  assumes  respon¬ 
sibility.  All  updated  documentation,  source 
and  object  code  listings,  and  all  otlier  items 


or  proeedurcs.  Some  questions  whose 
answers  will  provide  tailoring  guidance  are: 

(a)  Is  all  of  the  documentation  deseribed  in 
DOD-STL)-2  lf)7A  nece.ssary? 

(b)  What  documentation  is  alretidy  available 
even  if  in  a  different  format? 

(c)  Is  it  cost-effective  to  modify  it? 

(d)  Is  the  contractor’s  format  acceptable? 
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(e)  How  many  copies  are  actually  needed? 

(f)  How  can  DOD-STD-2167A  be  tailored? 

(g)  Is  a  formal  design  review  necessary  for 
each  CSCI? 

(h)  How  should  they  be  scheduled? 

DOD-STD-2167A  should  be  tailored  by 
deleting  non-applicable  requirements.  How 
does  a  program  manager  determine  which 
requirements  are  not  applicable?  Figure  5-10 
illustrates  the  tailoring  process. 

Most  tailoring  is  implemented  through  the 
statement  of  work  (SOW).  A  thorough  under¬ 
standing  of  requirements  (functional,  perfor¬ 


mance,  test,  documentation)  is  required  in 
order  to  properly  tailor  the  standards  and 
specifications. 

The  first  step  is  to  ask  if  the  requirement  is 
appropriate?  If  not,  then  tailor  it  out  through 
the  SOW.  If  the  requirement  is  appropriate, 
then  ask  if  the  requirement  is  adequate?  If  it 
is,  then  impose  the  requirement  through  the 
SOW.  If  the  requirement  is  not  adequate,  ask 
if  the  requirement  is  too  restrictive  or  too 
flexible?  If  it’s  too  restrictive,  delete  it  or 
modify  it  in  the  SOW.  If  it’s  too  flexible,  add 
to  or  modify  the  requirement  in  the  SOW. 
Use  careful  judgment  when  tailoring  a  pro¬ 
gram.  Don’t  arbitrarily  tailor  areas  simply  to 
reduce  program  costs.  In  the  long  run,  this 
may'increase  life  cycle  costs. 
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5.7  SUMMARY 

Software  that  is  part  of  a  weapon  system  is 
managed  by  partitioning  into  CSCIs.  Uach 
CSCI  is  managed  individually  and  follows  its 
own  software  development  cycle.  Software 
development  activities  can  be  broken  down 
into  six  steps;  any  of  which  can  be  repeated  as 
many  times  as  necessary  during  the  develop¬ 
ment  cycle.  These  six  steps  are  Software  Re¬ 
quirements  Analysis,  Preliminary  Design, 
Detailed  Design,  Coding  and  CSU  Testing, 
CSC  Integration  and  Testing,  and  CSCI  Test¬ 
ing.  These  steps  typically  occur  during  the 
Full  Scale  De/elopment  Phase. 

DOD-STD-2167A  is  the  approved  standard 
to  be  used  by  DOD  agencies  for  software 
development.  It  is  to  be  used  in  conjunction 
with  DOD-STD-2168,  Software  Quality  Pro¬ 
gram  Plan.  These  two  standards  are  not  in¬ 
tended  to  discourage  the  use  of  any  particular 
software  development  method,  but  instead,  to 
aid  the  Program  Manager  in  developing  and 
maintaining  quality  software.  They  should  be 
used  throughout  the  acquisition  life  cycle  and 
tailored  according  to  system  needs. 

It  is  especially  important  to  develop  the 
product  as  a  system.  Never  lose  sight  of  the 
fact  that  hardware  and  software  development 
are  intimately  related.  Although  they  arc 
developed  in  parallel,  software  is  almost  al¬ 
ways  in  the  critical  path  and  it  is  up  to  the 


Program  Manager  to  insure  proper  integra¬ 
tion  of  the  two  through  carefully  planned 
reviews  and  audits.  The  talents  of  an  inde¬ 
pendent  verification  and  validation  (IV&V) 
activity  may  be  used  to  aid  in  this  process. 
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CHAPTER6 


SOFTWARE  TEST  AND  EVALUATION 


Software  test  and  evaluation  are  two  of  the 
most  difficult,  frustrating,  and  expensive  ac¬ 
tivities  that  are  performed  during  system 
development.  Unfortunately,  they  may  also 
be  the  most  misunderstood  functions  of  the 
entire  system  acquisition  cycle.  Before  dis¬ 
cussing  the  details  of  software  testing,  let  us 
review  the  software  development  life  cycle 
and  see  how  the  software  test  process  fits  in. 

6.1  TEST  PLANNING 

Test  and  Evaluation  (T&E)  planning  is  in¬ 
itiated  at  the  inception  of  the  development 
process.  During  the  Concept  Exploration 
(CE)  Phase  the  initial  draft  of  the  Test  and 
Evaluation  Master  Plan  (TEMP)  is 
developed.  The  TEMP  is  the  basic  planning 
document  for  all  T&E  related  to  a  particular 
system  acquisition  and  is  used  by  the  Office  of 
the  Secretary  of  Defense  (OSD)  and  all  DOD 
components  in  planning,  reviewing,  and  ap¬ 
proving  all  T&E  activities.  The  TEMP 
provides  the  basis  and  authority  for  all  other 
detailed  T&E  planning  documents. 


The  TEMP  addresses  tv/o  types  of  computer 
resources:  system  support  and  embedded 
computer  resources. 

6.1.1  System  Support  Computer  Resources 

System  support  computer  resources  include 
all  the  government  and  contractor  planned 
software  and  computer  resources,  required  to 
fully  test  the  overall  system.  These  test 
resources  include  [1]: 

Test  Support  Equipment  -  All  unique  or 
modified  test  support  equipment  required  to 
conduct  the  planned  test  program  including 
any  special  calibration  and  software  require¬ 
ments. 

Threat  Systems  -  All  threat  simulators  against 
which  the  system  will  be  tested  including  the 
number  and  timing  requirements. 

Simulators,  Models,  and  Testbeds  -  All  sys¬ 
tem  simulations  required  including  com¬ 
puter-driven  simulation  models  and 
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hardware-in-the-loop  testbeds  identified  by 
specific  test  phase. 

Special  Requirements  -  All  non-instrumenta¬ 
tion  capabilities  and  resources  required  such 
as  special  data  processing  or  databases. 

For  all  of  these  test  resources  the  system  re¬ 
quirements  are  compared  with  existing  and 
programmed  capabilities  in  order  to  identify 
any  major  shortfalls. 

6.1.2  Mission  Critical  Computer  Resources 

The  initial  draft  of  the  TEMP  will  include  a 
preliminary  Software  Test  and  Evaluation 
Plan  in  Part  III.  This  plan  describes  the  an¬ 
ticipated  software  testing  necessary  to 
demonstrate  the  ability  of  the  mission  critical 
computer  resources  to  achieve  the  system  ob¬ 
jectives.  .  Detailed  information  on  the  TEMP 
may  be  found  in  DOD  Directive  5()()0.3-M-3, 
Test  and  Evaluation  Master  Plan  (TEMP) 
Guidelines,  26  Jan  1990. 

During  the  Demonstration/Validation  (D/V) 
Phase,  the  TEMP  is  updated  to  reflect  further 
refinements  in  the  objectives  and  evaluation 
criteria  of  the  weapon  system  computer 
resources  and  to  include  plans  for  Develop¬ 
mental  Test  and  Evaluation  (DT&E)  and 
Operational  Test  and  Evaluation  (OT&E). 
As  part  of  contractor  involvement,  a  Software 
Development  Plan  (SDP)  is  generated  along 
with  a  Software  Test  Plan  (STP).  The  contrac¬ 
tor  developed  STP  must  reflect  the  overall 
Software  Test  and  Evaluation  Plan  as  stated 
in  the  TEMP.  Along  with  the  SDP,  they  be¬ 
come  the  basic  documents  governing  the  con- 
duct  of  the  mission  critical  computer 
resources  development  and  test  activities. 

Once  the  Software  Requirements  Specifica¬ 
tion  (SRS)  and  the  Interface  Requirements 
Specification  (IRS)  have  been  generated  and 


approved,  the  preliminary  software  design  is 
initiated  (Figure  6-1).  Preliminary  design  is 
the  development  of  an  overall  skeletal  struc¬ 
ture  or  architecture  for  the  software.  The 
overall  structure  is  defined  to  include  such 


Fig.  6-1  Preliminary  Design 


things  as  the  types,  names  and  number  of 
software  modules;  their  calling  sequence  and 
their  input  and  output  parameters;  their  ap¬ 
proximate  execution  times;  and  other  per¬ 
tinent  relationships  that  should  exist  between 
the  various  modules.  Since  Ada  modules  and 
packages  can  be  compiled  without  the  re¬ 
quirement  that  lower  level  code  be  available, 
preliminary  design  using  an  Ada-based  Pro¬ 
gram  Design  language  (PDL)  is  very  valu¬ 
able.  Please  note  that,  if  a  technique  such  as 
Object  Oriented  Design  is  used,  the  proce¬ 
dure  just  described  may  not  really  apply.  In¬ 
stead  objects  and  classes  are  established  along 
with  a  clear  definition  of  data  requirements. 

Official  government  approval  will  be 
provided  once  a  preliminary  design  has  been 
completed  and  a  Preliminary  Design  Review 
(PDR)  has  been  held.  The  PDR  should  be 
approached  with  care  and  preparation  since 
its  completion  is  a  signal  to  the  contractor  to 
proceed  with  the  detailed  design. 
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During  the  detailed  design  (Figure  6-2),  the 
overall  architecture  developed  during  the 
preliminary  design  is  fleshed  out  with  detailed 
algorithms  and  logic  implementation  details. 
Figure  6-2  shows  the  detail  process  and  data 
flow  of  a  typical  task  using  Buhr  structure 
graphs  [2].  In  the  past,  pseudo-code  (English- 
like  programming  statements)  and  traditional 
flowcharts  were  often  used  in  this  phase. 
DOD  policy,  however,  now  requires  that  the 
detailed  design  also  be  developed  using  Ada 


group  computer  software  component  (CSC) 
testing  can  start.  This  testing  is  usually  infor¬ 
mal  in  nature  and  is  usually  performed  by  the 
programmer.  Formalized  testing,  as 
described  in  the  Software  Development  Plan 
and  the  Software  Test  Plan,  can  begin  after 
the  completion  of  coding  and  informal 
programmer  testing.  The  contractor  can  con¬ 
duct  his  own  formalized  testing  but  any  formal 
testing  which  is  to  be  witnessed  by  the  govern¬ 
ment,  doesn’t  begin  until  a  formal  Test  Readi¬ 


er  an  Ada  based  PDL  Once  again,  use  of  Ada 
as  a  design  language  will  be  immensely  useful 
because  the  actual  coding  of  the  detailed 
design  will  be  a  natural  follow-on  to  the 
detailed  design.  During  both  the  preliminary 
and  detailed  design  phases,  software  practices 
such  as  inspections  and  walk-throughs  (which 
will  be  discussed  in  detail  later  in  this  chapter) 
will  be  immensely  useful  in  finding  errors  or 
inconsistencies  in  the  overall  system  design. 

Coding  initiates  the  process  of  building  sub¬ 
modules.  These  sub-modules  are  progres¬ 
sively  combined  with  other  sub-modules  to 
form  larger  and  more  complex  modules  and 
blocks  of  software  (Figure  6-3).  With  each 
submodule  or  unit  coded,  individual  and 


ness  Review  (TRR)  has  been  conducted.  The 
purpose  of  the  TRR  is  to  determine  whether 
the  contractor  has  completed  his  own  testing 
and  has  the  resources  and  the  plans  and  pro¬ 
cedures  to  formally  demonstrate  to  the 
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Fig,  6-3  Coding 
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government  that  the  software  works  as  an 
entity. 

Software  integration  testing  exercises  a  single 
Computer  Software  Configuration  Item 
(CSCI).  This  type  of  testing  usually  requires 
a  dedicated  mainframe  computer  since  other 
software  or  subsystem  simulations  will  have  to 
be  used  (Figure  6-4).  The  actual  hardware 
typically  is  not  available  at  this  time  and  a 
simulation  or  emulation  is  substituted.  The 
target  computer  simulation  is  called  an  inter¬ 
pretive  computer  simulation  (ICS).  One  of 
the  primary  purposes  of  this  detailed  software 
system  testing  is  to  ensure  that  the  software  is 
inherently  sound  and  that  it  demonstrates  the 


potential  for  performing  its  function  once  it  is 
married  to  the  system  hardware  in  a  test  setup 
called  a  hot  bench. 

Hot  bench  integration  or  testing  is  perhaps 
one  of  the  most  frustrating  parts  of  software 
T&E  (Figure  6-5).  Once  the  software  is  mar¬ 
ried  to  the  actual  hardware,  the  difficult  part 
begins.  The  first  step  is  to  establish  a  hot 
bench  or  Systems  Integration  Lab  (SIL),  as  it 
is  sometimes  called.  The  next  step  is  to  popu¬ 
late  the  SIL  with  actual  black  boxes,  cable 
runs,  power  supplies,  displays  and  system 
computers,  configured  as  closely  as  possible 
to  the  final  article.  Unfortunately,  a 


Fig.  6-5  Hot  Bench  Integration 


laboratory  environment  can  only  ap¬ 
proximate  the  real  world  so  a  laboratory  is  a 
relatively  artificial  and  benign  environment. 
In  spite  of  this,  hot  bench  testing  is  the  key  to 
proving  that  the  hardware  and  software 
designs  satisfy  system  requirements. 

System  DT&E  and  OT&E  can  begin  once  hot 
bench  testing  is  successfully  completed  (Fig¬ 
ure  6-6).  Although  the  two  tests  are  some¬ 
times  combined  because  of  schedule 
constraints,  the  objectives  of  the  tests  are  dif¬ 
ferent  and  the  kinds  and  amount  of  data  re¬ 
quired  by  each  is  different.  Usually  the  level 
of  detail  required  for  DT&E  is  much  more 
than  that  required  for  OT&E.  This  means  that 
the  hardware  and  software  testing  required 


Fig.  6-4  Software  Integration  Testing 
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for  DT&E  is  usually  more  stringent  and  more 
extensive  than  that  required  for  OT&E. 

6.2  COST  OF  SOFTWARE  FIXES 

I 

Before  discussing  the  details  of  software  test¬ 
ing,  it  would  be  instructive  to  examine  the  cost 
of  removing  errors  from  software.  Figure  6-7 
shows  the  cost  of  a  typical  software  develop¬ 
ment  project  broken  down  by  the  various 
phases  of  development  [3].  The  cost  of  finding 
and  correcting  a  software  problem  in  the  early 
phases  is  insignificant  when  compared  to  the 
cost  of  finding  and  correcting  the  same  prob¬ 
lem  once  the  software  has  been  delivered. 
Although  these  are  not  absolute  numbers  that 
apply  all  development  projects,  the  message 
is  clear;  "Spend  more  time  up  front  finding 

’  errors  and  you  will  reduce  your  overall  cost." 


(b)  The  amount  of  paperwork  and  the  amount 
of  detail  to  be  examined  is  much  less.  Top- 
level  design  requirements,  interfaces,  and  test 
requirements  are  being  examined  without 
regard  to  implementation  details. 

(c)  Errors  introduced  at  the  top  level,  such  as 
in  the  requirements  or  overall  system  design, 
will  be  propagated  manifold  into  the  detailed 
design  and  coding  of  the  discrete  software 
components. 

(d)  Software  programmers  seem  to  undergo  a 
psychological  change  once  code  has  been 
generated  and  computer  based  testing  has 
begun  [4].  They  seem  to  be  less  sensitive 
about  their  mistakes  when  dealing  with  re¬ 
quirements  and  design  considerations  than 
they  are  about  errors  in  coding. 


A  question  that  may  be  asked  is,  "Why  does  it 
cost  so  much  less  to  find  and  correct  errors  in 
the  earlier  stages  of  software  development 
than  it  does  after  .system  delivery?"  Several 
reasons  that  have  been  put  forth: 

(a)  At  the  beginning  the  pressure  is  less  in¬ 
tense.  Since  there  is  little  or  no  code  to  ex¬ 
amine,  there  is  more  time  to  look  for  the 
problem. 


(e)  Major  resources  are  tied  up  in  testing  once 
computer  based  testing  begins.  Mainframe 
computers,  weapon  .system  hardware,  tech¬ 
nicians,  and  .system  analysts  can  escalate  cost 
very  quickly.  Once  a  problem  is  found  there  is 
tremendous  p.sychological  pressure  to  correct 
it  as  soon  as  possible.  Unfortunately,  the 
amount  of  paperwork  involved  in  the  correc¬ 
tion  procedure  is  immense  and  the  impact  of 
one  software  error  usually  propagates 
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throughout  other  modules.  Corrections  are 
not  always  complete  or  totally  accurate  [4], 

In  summary,  it  is  very  clear  that  the  greatest 
return  on  the  dollars  invested  in  finding  and 
correcting  software  problems  occur  during 
the  early  stages  of  software  development:  the 
requirements  and  design  phases. 

63  SOURCES  OF  SOFTWARE  ERRORS 

If  one  examines  where  the  typical  software 
errors  occur,  it  is  somewhat  surprising  to  learn 
that  they  also  occur  mostly  in  the  early  stages 
of  software  development.  As  can  be  seen 
from  Figure  6-8,  about  40%  of  all  software 
errors  are  attributed  to  problems  in  specifica¬ 
tion.  Twenty-eight  percent  are  due  to  incom¬ 
plete  or  erroneous  specifications  and  12%  are 
due  to  intentional  deviations  from  the 
specification.  Violations  of  programming 
standards  contribute  another  10%.  Errors 
due  to  coding  and  programming  mistakes 
(i.e.,  erroneous  data  accessing,  erroneous 
logic,  erroneous  computations,  improper  in¬ 
terrupts,  wrong  constants  and  data  values) 
comprise  only  about  38%  of  all  errors  [5]. 
Comparison  of  this  figure  with  the  previous 
figure  is  very  revealing.  It  tells  us  that  the  bulk 
of  the  errors  occur  during  those  phases  of 
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software  development  when  errors  are  the 
least  expensive  to  fix.  The  conclusion  is  very 
obvious:  if  more  time  and  effort  are  put  into 
requirements  definition  and  design,  fewer 
mistakes  will  be  made  and  those  that  are  made 
will  be  cheaper  to  correct.  The  more  you  pay 
now,  the  less  that  you  will  pay  later. 

6.4  TYPES  OF  TESTING 

Software  testing  can  be  broken  up  into  three 
general  categories:  human  testing,  software 
only  testing,  and  integration  testing. 

6.4.1  Human  Testing 

Human  testing  is  defined  as  an  informal,  non- 
computer-based  method  of  examining  com¬ 
puter  program  architectures,  designs  and 
internal  and  external  interfaces  for  the  ex¬ 
press  purpose  of  determining  how  well  they 
reflect  overall  system  requirements  [4]. 
Human  testing  is  comprised  of  inspections, 
walk-throughs,  desk  checking,  peer  ratings, 
and  design  reviews. 

6.4.1. 1  Inspections 

During  an  inspection,  the  programmer  ex¬ 
plains  to  a  group  of  three  or  four  peers  the 
overall  approach,  rationale,  choice  of  algo- 
rithm.s,  logic,  and  overall  module  structure  of 
the  program.  The  purpose  is  to  ensure  cor¬ 
rectness  and  consistency  in  structure,  coding 
conventions  such  as  variable  definitions  and 
u.se,  programming  standards  and  procedures, 
and  overall  unity  of  design.  At  the  end  of  the 
inspection,  which  usually  lasts  about  two 
hours,  all  errors,  inconsistencies,  and  omis¬ 
sions  are  listed  and  given  to  the  programmer 
for  correction.  Under  no  circumstances  is  the 
list  of  errors  ever  allowed  to  be  reviewed  by 
the  programmer’s  supervis  ,  or  anyone  else. 
The  purpose  of  the  inspection  is  to  improve 
the  final  product  in  a  non-threatening  en- 
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vironment  and  not  to  appraise  the 
programmer’s  performance.  Statistics  are 
also  collected  in  order  determine  the  quality 
of  the  product  and  the  progress  being  made  in 
the  development  process. 

6.4.1.2  Walk-throughs 

During  a  walk-through,  the  group  of  three  or 
four  peers  come  prepared  with  test  cases  so 
that  they  can  "play  computer"  and  mentally 
step  through  the  design  and  the  logic  flow.  As 
is  done  in  inspections,  all  errors,  inconsisten¬ 
cies,  and  omissions  are  summarized  and  given 
to  the  programmer  for  correction.  The  results 
are  also  confidential  and  are  never  seen  by 
anyone  outside  the  group,  including  the 
programmer’s  supervisor.  The  climax  of  both 
inspections  and  walk-throughs  is  a  meeting  of 
the  minds  between  all  the  participants  [4]. 
Statistics  may  once  again  be  gathered. 

6.4.1.3  Desk  Checking 

Desk  checking  is  the  least  productive  of  all 
human  testing  since  it  involves  the  program¬ 
mer  sitting  at  his  desk  and  reviewing  his  work. 
It  is  human  nature  to  miss  errors  that  one  has 
committed.  Desk  checking,  however,  can  be 
performed  individually  at  any  time  and 
without  the  need  for  convening  a  meeting 
with  other  individuals.  It  is  better  than  doing 
nothing  at  all. 

6.4.1.4  Peer  Ratings 

Peer  ratings  involve  a  group  of  programmers 
reviewing  each  other’s  work.  Each  program¬ 
mer  anonymously  submits  one  or  two 
modules  for  review.  The  peer  group  reviews 
each  module  and  rales  the  overall  quality 
while  suggesting  improvements.  Results  are 
tabulated  and  passed  around.  Like  inspec¬ 
tions  and  walk-throughs,  peer  reviews  are 


very  beneficial  in  improving  commonality, 
overall  consistency,  and  program  integrity. 

There  are  other  benefits  to  inspections,  walk¬ 
throughs,  and  peer  ratings  beside  the  obvious 
one  offinding  errors  and  inconsistencies.  The 
participants  themselves  benefit  because  they 
are  exposed  to  other  programming  styles  and 
new  techniques  which  they  may  want  to  adapt 
for  their  own  use. 

6.4.1. 5  Design  Reviews 

PDRs  and  CDRs  are  not  normally  considered 
by  the  government  as  part  of  the  test  process. 
This  is  unfortunate  because,  to  the  contractor, 
completion  of  the  PDR  is  a  signal  that  the 
preliminary  design  is  acceptable  and  that  he 
can  proceed  with  the  detailed  design. 
Likewise,  completion  of  the  CDR  is  the 
contractor’s  signal  that  the  detailed  design  is 
acceptable.  Very  often  the  government  fails 
to  perform  a  thorough  review  of  all  the 
documentation  submitted  prior  to  either  the 
PDR  or  the  CDR.  PDRs  and  CDRs  may  be 
conducted  superficially  and  the  contractors 
may  proceed  into  detailed  design  without  a 
thorough  review  of  the  overall  design  and 
without  assurances  that  overall  system  re¬ 
quirements  are  reflected  in  the  design. 

There  is  no  immediate  solution  to  this  prob¬ 
lem  since  it  is  often  due  to  the  lack  of  adequate 
manpower.  Documentation  review  prior  to 
design  reviews  may  always  be  inadequate.  Al¬ 
ternatives  are  to  thoroughly  review  only  criti¬ 
cal  portions  of  the  design,  thoroughly  review 
only  the  module  interfaces,  or  augment  the 
engineering  staff  with  outside  help.  If  the 
funding  is  available,  an  Independent  Verifica¬ 
tion  and  Validation  (IV&V)  contractor  can  be 
u.sed.  Outside  consultants  can  be  brought  in, 
or  heip  can  be  sought  from  the  various 
military  labs  or  software  support  agencies. 
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Another  solution  is  to  hold  incremental  PDRs 
and  CDRs.  By  spreading  out  the  reviews  the 
available  staff  can  pay  more  attention  to 
details.  This,  of  course,  has  to  be  weighed 
against  overall  schedule  slippage.  One  must 
remember,  however,  that  an  incompletely 
reviewed  software  design  is  guaranteed  to  in¬ 
troduce  schedule  slippage  anyway.  As  the 
commercial  states,  "You  can  pay  me  now,  or 
you  can  pay  me  later." 

6.4.1. 6  BcncHts  of  Human  Testing 

Human  testing  is  a  very  productive  undertak¬ 
ing.  In  his  book.  The  Art  of  Software  Testing, 
Myers  [3]  discusses  the  following  positive 
qualities  of  human  testing: 

(a)  Experience  has  shown  that  it  is  quite  effec¬ 
tive  in  finding  errors  and  it  should  be  used  on 
every  programming  effort; 

(b)  Since  it  is  usually  applied  between  the  start 
of  design  and  the  beginning  of  computer 
based  testing,  it  substantially  contributes  to 
productivity  and  reliability; 

(c)  It  may  find  30  to  70%  of  design  and  coding 
errors; 

(d)  There  is  a  higher  probability  of  proper 
error  correction  since  they  are  found  early  in 
the  development  phase.  Programmers  tend 
to  make  more  errors  correcting  errors  found 
during  computer-based  software  testing; 

(e)  It  finds  errors  in  clusters  or  batches  as 
opposed  to  computer-based  testing  which 
finds  errors  one  at  a  time; 

(f)  It  lowers  the  cost  of  software  testing  since 
the  costs  involved  are  a  few  programmer’s 
man-hours  as  opposed  to  the  computer 
resources  and  the  large  number  of  personnel 
involved  in  computer-based  testing. 


In  summary,  human  testing  provides  the 
highest  return  on  your  investment  of  valuable 
software  test  and  evaluation  dollars. 

6.4.2  Software  Only  Testing 

Software  only  testing  is  defined  as  that  testing 
that  is  performed  solely  for  the  purpose  of 
determining  the  integrity  of  the  software 
when  it  is  tested  as  an  entity.  In  other  words, 
one  wants  the  assurance  that  the  software 
works  before  it  is  married  to  the  hardware. 
The  reason  for  this  is  that  software  is  usually 
integrated  with  newly  developed  hardware 
which  has  its  own  maturation  problems.  In¬ 
tegrating  untested  software  with  unproven 
hardware  makes  it  very  difficult  to  determine 
where  the  problems  lie:  hardware  or  software. 

6.4.2.1  Black  Box  or  Functional  Testing 

Black  box  testing  entails  testing  a  particular 
software  unit  without  any  knowledge  about 
the  internal  structure  or  logic  of  that  unit 
(Figure  6-9).  Various  test  cases  are  generated 
based  on  the  specification  and  the  require¬ 
ments  of  the  unit.  The  correctness  of  the 
software  unit  is  determined  only  by  the  output 
data  generated  in  response  to  the  input  data. 
This  is  done  by  designing  the  test  cases  with 
two  types  of  input  data.  The  first  type  of  input 
data  falls  within  the  boundaries  expected  by 
the  software  unit  and  the  second  type  of  data 
falls  outside  these  boundaries.  The  reason  for 


Fig.  6-9  Black  Box  or  Functional  Testing 
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the  latter  is  that  oftentimes  it  is  those 
parameters  (i.e.,  invalid)  that  do  not  fall 
within  the  expected  limits  that  cause  the  big¬ 
gest  problems.  It  is  an  old  dictum  in  software 
development  that  ensuring  that  the  software 
doesn’t  do  the  unexpected  is  as  important  as 
ensuring  that  it  does  what  it  is  supposed  to  do. 

The  primary  objective  of  software  testing  is  to 
demonstrate  performance  and  reliability. 
Often  it  is  useful,  if  not  essential,  to  approach 
software  testing  (particularly  at  the  lower 
levels)  with  the  attitude  of  "Let’s  try  to  break 
it."  This  is  to  demonstrate  the  ability  of  the 
software  to  recover  from  abnormal  events  or 
to  degrade  in  a  graceful  or  controlled  manner. 
Since  complete  and  exhaustive  testing  of  any 
practical  size  software  is  impossible,  one  must 
develop  an  acceptable  level  of  maturity  in  the 
software.  This  is  often  measured  by  the  num¬ 
ber  of  faults  that  are  occurring  during  te.st.  As 
software  matures,  the  number  of  faults  dis¬ 
covered  decreases  and,  as  a  minimum,  the 
software  becomes  controllable  (i.e.,  the  con¬ 
figuration  is  stable). 

The  reason  that  exhaustive  testing  of  software 
is  a  practical  impossibility  is  that  the  number 
of  valid  and  invalid  test  cases  is  infinite.  For 
example,  if  a  certain  input  parameter  should 
fall  within  the  values  of  1  and  10,  truly  exhaus¬ 
tive  testing  would  test  all  the  values  between 
1  and  10.  This  is  clearly  an  impossibility  since 
there  are  an  intinite  number  of  valid  values 
between  these  two  limits.  Likewise,  exhaus¬ 
tive  testing  of  all  invalid  p'  rameters  is  also 
impossible  because  these  are  also  infinite. 

To  summarize,  software  of  any  practical  size 
can  never  be  exhaustively  tested  and  can 
never  be  guaranteed  to  be  totally  error  free. 
Because  of  this,  the  goal  of  software  testing  is 
to  demonstrate  a  certain  level  of  performance 
so  that  there  are  reasonable  assurances  that 
the  software  performs  according  to  the 


specification.  How  reasonable  the  assurances 
are  is  a  simple  matter  of  economics,  how 
much  can  one  afford  to  test? 


6.42.2  White  Box  or  Structural  Testing 


Unlike  black  box  testing,  white  box  testing  is 
concerned  with  the  internal  structure  and 
logic  of  the  software  unit.  The  test  cases  are 
generated  using  a  listing  of  the  code  as  op¬ 
posed  to  the  requirements  specification  (Fig¬ 
ure  6-10).  Generation  of  the  input  data, 
however,  is  very  similar  to  the  generation  of 
input  data  for  black  box  testing.  Both  valid 
and  invalid  input  data  are  generated  for  the 
te.st  cases  and  the  output  data  is  analyzed  to 
determine  the  correctness  of  the  computer 
code.  Exhaustive  white  box  testing  is  also 
prohibitive  for  the  same  reasons  that  exhaus¬ 
tive  black  box  testing  is  impossible. 


Exhaustive  white  box  testing,  however,  does 
have  another  dimension  other  than  all  the 
po.ssible  valid  and  invalid  input  parameters. 
Since  the  tester  has  knowledge  of  the  internal 
logic  and  program  structure,  a  goal  would  be 
to  test  all  possible  logic  paths  that  exist  within 
the  code.  Although  all  paths  can  be  executed, 
it  is  impo.ssible  to  test  all  combinations.  As  an 
example,  consider  the  logic  flow  of  Figure 
6- 1 1  which  shows  a  simple  logic  sequence  that 


INPUT 

DATA 


OUTPUT 
-  DATA 


■ 


LISTING 


Fig.  6-10  White  Box  or  Structural  Testing 
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contains  two  loops  that  execute  up  to  12  times. 
This  sequence  contains  10  raised  to  the  20th 
power  different  finite  paths.  If  one  could  test 
one  logical  path  every  nanosecond  (trillionth 
of  a  second),  it  would  take  4000  years  to  test 
all  the  different  paths.  Using  a  real  example, 
it  would  take  over  60,000  years  to  perform  a 
similar  type  of  exhaustive  test  on  the  Titan  III 
missile  guidance  software. 

The  goal  of  white  box  testing  then  is  also  to 
minimize  the  number  of  errors  in  the 
delivered  code  and  to  provide  reasonable  as¬ 
surances  that  the  software  performs  accord¬ 
ing  to  the  specification. 

6.4.2J  Top-Down/Bottom-Up  Testing 

In  actual  practice  the  best  course  of  action  is 
to  develop  a  software  test  strategy  that  incor¬ 
porates  both  black  box  and  white  box  testing 
philosophy.  Test  procedures,  therefore, 
should  make  use  of  both  the  specifications 
and  the  existing  listing.  Regardless  of  what 
test  philosophy  is  used,  the  choice  remains 
whether  to  perform  bottom-up  testing  or  top- 
down  testing. 


"drivers"  (Figure  6-12).  These  drivers  are 
coded  and  used  for  the  specific  module  test 
and  then  replaced  when  the  next  higher  level 
module  is  coded  and  integrated. 


Top-down  testing  begins  by  testing  the 
highest  level  module  first  and  then  progres¬ 
sively  integrating  and  testing  lower  level 


DRIVER 


Fig.  6-12  Bottom-Up  Testing 


modules.  A  higher  level  module  requiring 
lower  level  modules  uses  dummy  modules 
that  are  called  "stubs"  (Figure  6-13).  The  stub 
may  be  as  simple  as  sending  a  message  such  as 
'This  stub  is  a  replacement  for  the  sorting 
module"  or  as  complicated  as  performing 
dummy  commands  to  simulate  actual  process¬ 
ing  times. 


Bottom-up  testing  begins  with  the  lowest  level 
module  or  unit  (i.e.,  one  that  does  not  call  any 
other  module  or  unit)  and  tests  them  through 
the  use  of  dummy  modules  or  units  called 
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Fig.  6-11  Software  Complexity 


There  are  benefits  and  problems  associated 
with  both  approaches.  In  the  bottom-up  ap- 
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Fig.  6-13  Top-Down  Testing 


proach  a  substantial  amount  of  time  is  spent 
in  designing,  coding,  and  testing  module 
drivers.  Since  these  drivers  will  only  be  used 
for  testing,  it  is  somewhat  wasteful  of  a 
programmer’s  time.  In  addition  many  sig¬ 
nificant  timing  problems  may  be  masked  and 
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not  discovered  until  late  in  the  test  cycle  when 
both  schedules  and  resources  are  tight.  On 
the  plus  side,  if  the  driver  was  properly 
designed,  the  next  higher  level  module  can  be 
tested  with  some  assurances  that  thf^  lower 
level  modules  are  correct. 

In  the  top-down  approach,  there  is  a  tendency 
to  code  the  easier  modules  first.  Often  com¬ 
plex  and  critical  modules  are  not  coded  and 
tested  until  late  in  the  test  cycle  because  of  the 
difficulty  in  predicting  module  complexity. 
These  modules  may  sometimes  require  more 
time  and  energy  than  was  initially  planned  or 
is  currently  available.  On  the  plus  side,  once 
a  higher  level  module  is  coded  and  tested,  it 
is  repeatedly  tested  as  more  and  more  stubs 
are  replaced  by  fully  functioning  modules. 
This  repeated  testing  will  test  the  internal 
consistency  of  each  module  and  its  interface 
with  the  rest  of  the  software.  It  will  also  reveal 
major  timing  problems  very  early  in  the  test 
cycle  while  there  are  sufficient  resources  to 
adequately  solve  major  problems.  This  evolu¬ 
tionary  approach  helps  to  build  up  maturity 
and  confidence  in  the  software. 

In  a  manner  analogous  to  black  box  and  white 
box  testing,  the  testing  philosophy  should  in¬ 
corporate  both  top-down  and  bottom-up  test¬ 
ing.  The  most  complex  and  critical  modules 
may  be  designed  and  coded  from  the  bottom- 
up  and  the  rest  of  the  system  can  be  designed 
and  tested  from  the  top-down.  The  key  is  to 
accurately  determine  the  critical  and  complex 
modules;  this  is  not  always  an  ea.sy  task. 

6.4.2.4  Software  System  Testing 

Integration  and  testing  of  the  software  as  an 
entity  can  begin  once  all  the  modules  in  the 
configuration  item  have  been  coded  and  in¬ 
dividually  tested  by  the  programmer.  When 
you  test  the  software  as  a  system,  you  are  not 
trying  to  duplicate  the  results  of  the  various 


lower  level  tests.  The  main  purpose  of  the 
software  system  test  is  to  ensure  that  the 
coded  software,  meets  the  overall  perfor¬ 
mance  objectives  of  the  software  require¬ 
ments  and  .system  specifications.  In  other 
words,  does  the  assembled  software  package 
meet  the  overall  system  requirements? 

In  order  to  conduct  software  system  tests,  it  is 
necessary  to  use  the  target  computer  or  to 
emulate/simulate  it  in  the  software  worksta¬ 
tion  or  mainframe.  One  must  also  simulate 
the  rest  of  the  hardware  and  the  external  en¬ 
vironment  (Figure  6-14).  In  addition  large 
amounts  of  data  may  have  to  be  generated  and 
introduced  into  the  software  according  to  ap¬ 
proved  test  plans  and  procedures.  Since  the 
simulations  required  are  sometimes  exten¬ 
sive,  the  software  tester  must  have  assurances 
that  the  simulations  are  correct.  This  may  not 
be  simple.  Lack  of  adequate  simulations  is  a 
major  contributor  to  delays  in  software  system 
testing.  If  the  program  is  sharing  computer 
resources  with  other  programs  or  organiza¬ 
tions,  allocation  of  computer  time  and 
priorities  must  be  carefully  considered.  Many 
programs  have  been  delayed  by  too  many  or¬ 
ganizations  vying  for  the  same  resources. 


Fig.  6-14  Software  System  Testing 

Purchasing,  installing,  and  checking  out 
another  computer  system  lakes  months  and 
leasing  time  on  an  off-site  computer  is  not 
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only  expensive  but  also  slows  clown  the  turn¬ 
around  time. 

6.4.3  Integration  Testing 

Upon  completion  of  software  system  testing, 
systems  developers  embark  on  perhaps  the 
most  difficult,  time-consuming  and  frustrat¬ 
ing  part  of  integrated  testing-hot  bench  test¬ 
ing. 

6.4.3. 1  Hot  Bench  Testing 

If  the  system  software  works  when  tested  as  a 
CSCI,  the  next  step  is  to  marry  it  to  actual 
.system  hardware  configured  in  a  test  setup 
known  as  a  hot  bench  (Figure  6-5).  Other 
common  names  for  the  hot  bench  are  systems 
integration  lab  (SIL)  or  systems  integration 
facility  (SIF).  The  objective  is  to  test  the 
software  in  a  laboratory  environment  which 
closely  approximates  the  actual  system  to  be 
fielded.  The  hot  bench  will  include  the  black 
boxes,  cable  and  wire  runs,  data  busses,  and 
any  ancillary  eciuipment  reciuired  to  test  the 
.system  (e.g.,  terminals,  printers). 

Once  you  start  testing  the  software  with  real 
hardware,  you  will  discover  that  the  real  world 
is  a  very  "noi.sy"  environment.  Problems  that 
are  due  to  external  causes  will  be  propagated 
throughout  the  hot  bench.  For  example,  a 
large  air  conditioning  compressor  kicks  in  or 
the  heavily  used  copying  machine  next  door 
induce  large  transients  into  your  equipment. 
Electrical  transients  in  digital  system  can  cre¬ 
ate  unwelcome  behavior.  Sensitive  instru¬ 
ments  are  affected  by  stray  radio  frequency 
(RF)  signals  and  computers  do  unpredictable 
things  when  the  humidity  climbs  above  a  cer¬ 
tain  threshold  level.  Couple  this  with  the 
traditional  difficulty  encountered  in  deter¬ 
mining  whether  a  problem  is  caused  by  the 
hardware  or  the  software  and  one  soon  real¬ 
izes  that  an  inordinate  amount  of  time  iss[)ent 


chasing  ghosts,  random  malfunctions,  and 
other  elusive  spirits. 

Since  this  testing  usually  takes  place  towards 
the  end  of  Full  Scale  Development  and  coin¬ 
cides  with  the  initial  phase  of  Produc¬ 
tion/Deployment,  test  personnel  are  vying  for 
resources  with  various  other  groups.  This 
means  that  the  available  black  boxes  are  in 
demand  by  the  production  people,  who  are 
attempting  to  build  the  systems,  the  Automat¬ 
ic  Test  Equipment  (ATE)  folks,  who  are 
building  the  automated  software  to  test  these 
black  boxes,  and  the  test  personnel  conduct¬ 
ing  quality,  environmental  and  stress  tests. 

Obtaining  these  black  boxes  may  become  very 
difficult,  and  a  lack  of  adequate  backups  for 
failed  components  is  a  major  contributor  to 
test  delays.  Because  of  the  usually  high  cost 
of  these  items,  early  planning  for  these  assets 
will  not  solve  all  the  availability  problems. 
One  may  not  be  able  to  afford  to  buy  more  of 
the  hardware  retjuired.  A  compromise  is  to 
share  the  black  boxes.  Test  schedules  are 
often  driven,  not  by  the  complexity  of  the 
tests,  but  by  the  availability  of  resources. 
A.sset  management  is  an  important  aspect  of 
any  hot  bench  testing. 

Other  factors  which  must  be  considered  are 
the  facilities  and  the  scheduling  of  these 
facilities.  'Fo  perform  hot  bench  testing  the 
facility  must  have  adequate  power  and  air 
conditioning  to  handle  the  anticipated  power 
and  heat  loads.  TEMPFiST  requirements,  as 
well  as  other  security  requirements,  must  be 
adhered  to  if  the  testing  will  generate  or  use 
classified  data.  All  of  this  requires  careful 
planning  since  design  and  development 
schedules  may  change  and  the  facilities  may 
not  be  available  when  they  are  needed.  In 
addition,  if  the  hot  bench  testing  slips  too 
much,  conflicts  may  arise  with  other  programs 
which  may  have  planned  to  use  the  same 
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facility  space.  It  is  a  good  idea  to  plan  for  these 
unexpected  contingencies. 

Many  of  these  problems  can  be  minimized  if 
the  test  requirements  and  resources  are  ade¬ 
quately  addressed  in  the  TEMP  and  the 
Software  Test  Plan. 

6.4.3.2  DT&E/OT&E  Testing 

The  next  phase  of  testing  is  by  far  the  most 
expensive  of  all.  It  involves  performing  field 
tests  in  an  environmentclosely  approximating 
the  environment  of  the  anticipated  threat.  If 
flight  testing  is  involved,  one  must  include  the 
test  aircraft,  chase  aircraft,  test  ranges,  logis¬ 
tics  support  for  all  the  aircraft  and  test  equip¬ 
ment,  large  data  reduction  systems  and 
software  and  the  hundreds  of  personnel  re¬ 
quired  to  conduct  and  coordinate  the  various 
tests  (F'igure  6-6).  Clearly  hundreds  of 
thousands  to  millions  of  dollars  are  tied  up  in 
field  testing  major  systems. 

Logistic  support  for  DT&E  must  also  be 
planned.  Failure  rates  for  electronic  equip¬ 
ment  must  be  anticipated  and  provisions 
made  for  backups  and  repair  of  failed  com¬ 
ponents.  Repairs  may  be  done  organically  for 
OT&E  but  for  DT&E  the  repairs  are  usually 
performed  by  the  contractor. 

Scheduling  of  national  assets  such  as  the 
China  Uike  Naval  Weapons  Center  or  the 
Edwards  Flight  Test  Center  will  have  to  be 
done  years  in  advance  when  the  crystal  ball  is 
pretty  cloudy.  Schedule  uncertainties,  unpre¬ 
dictable  weather  and  unforeseen  technical 
problems  will  force  the  test  manager  to  con¬ 
stantly  generate  contingency  test  plans  to  ac¬ 
commodate  program  requirements  with 
those  of  other  legitimate  DOD  users  requir¬ 
ing  the  use  of  the  same  national  assets. 


Data  reduction  can  be  a  significant  bot¬ 
tleneck.  During  any  major  field  testing  ac¬ 
tivities,  prodigious  amounts  of  data  can  be 
generated.  Adequate  computer  resources 
and  sufficiently  tested  support  software  pack¬ 
ages  are  required.  It  is  not  uncommon  to  wait 
one  or  two  weeks  for  data  to  be  reduced  and 
presented  in  a  format  suitable  for  engineering 
analysis.  If  in  addition,  you  are  required  to 
have  organizations  other  than  your  own  (i.e., 
another  service  facility)  record  data  for  your 
test,  additional  bureaucratic  delays  can  fur¬ 
ther  add  time  to  your  test  program. 

The  most  important  thing,  however,  is  to  real¬ 
ize  that  field  testing  should  confirm  hot  bench 
results.  If  you’re  not  ready,  don’t  fly.  If  it 
doesn’t  completely  work  in  the  lab,  don’t 
waste  program  resources  using  the  field  as  a 
laboratory. 

6.5  TEST  TOOLS 

It  is  an  ironic  fact  that  the  very  industry  that 
has  automated  major  segments  of  other  in¬ 
dustries,  has  itself  failed  to  automate.  Thanks 
to  computers  and  software,  robots  and  auto¬ 
mation  are  major  players  in  such  diverse  in¬ 
dustries  as  automobile  manufacturing, 
publishing,  and  textiles.  When  it  comes  to 
software  development,  however,  it  is  all  vir¬ 
tually  manual  labor.  It  still  requires  a  person 
to  translate  a  set  of  requirements  into  a  set  of 
computer  instructions.  Computers  that 
generate  code  directly  from  human  languages 
are  still  far  in  the  future. 

This  doesn’t  mean  that  there  are  no  software 
development  tools.  Numerous  tools  exist 
and,  although  they  all  require  extensive 
human  participation,  they  are,  nevertheless, 
very  useful.  Test  tools  can  be  classified  into 
the  following  categories: 
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Requirements’  Analysis  Tools  -  These  tools 
provide  the  capability  to  incrementally  stale 
system  requirements  and  to  systematically 
check  for  consistency  and  completeness. 

Text  Editors  -  Tools  used  to  insert,  delete,  or 
manipulate  portions  of  any  text  such  as 
documentation,  lest  data  or  code. 

Test  Generators  -  Allow  test  drivers  to  be 
developed  in  a  systematic  and  standardized 
fashion.  Sometimes  can  be  automatically 
generated  during  the  coding  process. 

Static  Analysis  Tools  -  Perform  a  static 
analysis  of  a  i)articular  program  to  protluce  a 
listing  of  questionable  coding  practices, 
departures  from  programming  and  coding 
standards,  isolated  code,  symbol  tables,  etc. 

Debugging  Tools  -  On-line  tools  that  enable 
programmers  to  interact  with  their  program 
during  program  execution  in  order  to  assist 
them  in  locating  errors. 

Environmental  Simulators  -  Allow  for  the 
simulation  of  a  particular  system  environment 
when  testing  in  the  real  environment  is  either 
impractical  or  too  ex;  ensive. 

System  Simulators  -  Simulations  of  other  .sys¬ 
tems  which  interact  with  the  system  under 
development  or  simulations  of  major  subsys¬ 
tems  not  yet  available. 

Data  Reduction/Analysis  Fools  -  Software 
and  computer  packages  that  allow  large 
volumes  of  data  to  be  analyzed  and  reduced 
to  a  format  easily  digestible  by  engineers  and 
managers. 

6.6  DEBlJG(;iNG 

Debugging  is  the  activity  of  finding  and  cor¬ 
recting  a  known  error.  It  is  performed  after  a 


test  run  has  uncovered  errors.  Methods  for 
debugging  include:  brute  force,  induction, 
deduction,  backtracking,  and  testing  [4]. 

The  brute  force  method,  although  requiring 
little  thought,  is  the  most  inefficient  of  all 
debugging  methods.  This  is  because  it  usually 
takes  the  form  of  ( 1)  memory  dumps,  (2)  scat¬ 
tering  print  statements  in  the  code  or  (3)  using 
automated  debugging  tools  which  require 
labor  intensive  analysis  to  trace  the  error. 

Debugging  by  induction  requires  careful 
thought  and  follows  the  classical  inductive 
methotl  of  examining  the  data,  looking  for 
relationships,  devising  a  hypothesis  and  prov¬ 
ing  the  hypothesis  by  locating  the  error. 

Debugging  by  deductive  reasoning  proceeds 
from  general  theories  or  premises  about  the 
cause  of  an  error,  eliminates  and  refines  the 
premise.s,  arrives  at  a  conclusion  and  finds  the 
error. 

Backtracking  involves  starting  at  the  point 
where  the  incorrect  result  was  produced  and 
reverse  engineering  the  program  to  discover 
the  source  of  the  error. 

Debugging  by  testing  involves  the  generation 
of  small  samples  of  test  data  in  an  attempt  to 
isolate  the  problem. 

Of  all  programming  activities,  debugging  is 
probably  the  most  disliked.  This  is  because 
debugging  is  a  mentally  taxing  effort  and  quite 
often  it  is  very  damaging  to  sensitive  egos  that 
see  programming  errors  as  personal  failures 
or  indictments  of  their  programming  com¬ 
petence.  It  can  be  very  difficult  because  many 
faults  can  be  caused  by  errors  in  a  program 
statement  anywhere  in  the  entire  program. 
Developments  in  the  stale-of-practice  of 
software  engineering  are  minimizing  these 
types  of  errors. 
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There  are  certain  debugging  principles  that 
have  arisen  out  of  the  collective  experience  of 
programming  [4].  These  are: 

(a)  Be  careful  when  using  debugging  tools. 
They  often  introduce  more  problems  then 
they  solve  and  often  require  a  non-trivial  in¬ 
vestment  of  time  in  order  to  become  profi¬ 
cient  with  the  tool; 

(b)  Experience  has  shown  that  where  there  is 
one  bug,  there  are  usually  more  because  it  has 
been  shown  that  errors  tend  to  cluster; 

(c)  The  probability  of  a  fix  being  correct  is 
never  100%.  Correction  of  an  error  often 
introduces  additional  errors  requiring  further 
debugging; 

(d)  It  is  always  best  to  change  the  source  code 
and  not  the  object  code.  Correcting  the  object 
code  throws  the  source  code  and  the  object 
code  out  of  sync  and  may  produce  disastrous 
results  elsewhere  in  the  program  or  later  on 
in  the  execution; 

6.7  MANAGEMENT  GUIDANCE 

The  following  suggestions  provide  some 
general  guidance  during  the  planning  and  ex¬ 
ecution  of  software  test  and  evaluation. 

(a)  If  an  IV&V  organization  is  going  to  be 
used,  bring  them  on  board  early,  preferably 
during  the  Concept  Exploration  phase.  Just  a 
handful  of  people  (2-3)  performing  an  IV&V 
role  can  be  invaluable. 

(b)  When  evaluating  contractors’  proposals 
and  Software  Development  Plans  make  sure 
that  some  organization  other  than  the  one 
developing  the  software  performs  the  testing. 
The  developing  organization  may  develop 
test  plans,  but  it  should  not  test  its  own 


software.  This  independence  can  be  internal 
or  external  to  the  contractor; 

(c)  As  part  of  the  evaluation  process,  insure 
that  a  good  quality  assurance  organization  is 
in  place.  Since  software  development  is  so 
labor  intensive,  it  is  important  that  an  or¬ 
ganization  exists  to  impose  programming 
standards,  procedures  and  regimentation  on 
the  development  process  and  to  evaluate  both 
the  process  and  the  product; 

(d)  Perhaps  one  of  the  most  important  things 
to  remember  is  that  failure  to  impose  strict 
discipline  into  the  software  development 
process  is  the  quickest  way  to  ensure  disaster. 
This  discipline  must  exist  in  the  program  of¬ 
fice  as  well.  The  program  manager  must  be 
absolutely  resolute  in  insisting  that  a  software 
developer  complete  an  activity  before 
proceeding  to  the  next  one.  If  the  developer 
is  not  ready  for  a  PDR  or  CDR,  then  res¬ 
chedule  it.  If  they  htiven’t  finished  software 
testing,  then  don’t  proceed  into  hot  bench 
integration. 

(e)  The  program  office  should  have  the  even¬ 
tual  user,  the  supporting  activity,  and  the 
IV&V  contractor  (if  there  is  one)  participate 
in  the  thorough  review  of  the  Software  Test 
Plan.  'Phey  should  critically  examine 
whether: 

-  all  te.st  resources  have  been  identified; 

-  all  the  test  software  tools  are  available,  are 
matured  and  currently  in  use; 

-  the  test  schedule  is  adequate.  Remember 
that  software  development  is  almost  always 
on  the  critical  path; 

-  the  developer  really  understands  software 
testing  principles; 
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-  testing  is  complete  and  has  demonstrated 
operational  performance  and  reliability. 

(f)  Seek  help  at  the  very  beginning  of  the 
program  don’t  wait  until  software  develop¬ 
ment  is  in  trouble  because  then  it  is  too  late. 
It  doesn’t  hurt  to  have  an  ouiside  organization 
examine  the  status  of  the  software  at  critical 
intervals  during  the  development  cycle; 

(g)  Insist  that  program  office  personnel  attend 
software  training  courses  on  a  periodic  basis, 
especially  young  software  project  officers. 
Your  program  will  derive  benefits  by  allowing 
your  personnel  to  keep  up  with  software  prac¬ 
tice  tmd  technology. 

(h)  Make  absolutely  sure  that  the  program 
office  engineering  and  software  personnel 
perform  a  thorough  review  of: 

-  preliminary  software  design; 

-  test  plans  and  procedures; 

-  traceability  of  requirements  from  the  sys¬ 
tem  specification  all  the  way  down  to  the  test 
procedures; 

-  the  developer’s  configuration  management 
organization  and  procedures.  On  a  major 
software  development,  keeping  track  of  the 
various  versions  of  software  is  a  non-trivial 
task.  Many  major  test  programs  have  been 
severely  impacted  because  the  wrong  version 
of  software  has  been  tested. 

(i)  Neyer^iiever.  never  teat  suftwafe.lliaL.CDii- 
tains  patches!  A  patch  is  defined  as  a  piece  of 
machine  language  code  that  is  overwritten 
over  an  existing  piece  of  object  code.  Testing 
software  with  patches  is  nothing  more  than  a 
game  of  Russian  Roulette.  You  may  get  by 


once,  but  eventually  your  test  program  is 
going  to  blow  up  in  your  face!  Insist  that 
software  errors  be  corrected  at  the  source 
code  level.  Treat  error  correction  as 
redevelopment  by  tracing  all  the  way  back 
from  the  system  requirements  through  design 
and  by  ensuring  that  the  documentation  is 
updated. 

The  only  time  a  software  patch  is  justified  in 
a  software  test  program  is  when  it  is  necessaiy 
to  disable  a  function  for  safety  reasons  or 
when  it  is  impossible  to  conduct  a  particular 
test  without  the  use  of  a  patch.  At  any,  rate 
once  these  tests  are  over  get  rid  of  the  patch! 

Don’t  even  think  about  fielding  into  an  opera- 
tii)nal  environment  any  software  package  con¬ 
taining  patches! 
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CHAPTER? 


POST  DEPLOYMENT  SOFTWARE  SUPPORT 


7.1  BACKGROUND 

More  than  two  thirds  of  the  DOD’s  expendi¬ 
ture  for  software  is  for  Post  Deployment 
Software  Support  (PDSS)  [1].  The 
Electronics  Industry  Association  (El A)  has 
predicted  that  Mission  Critical  Computer 
Software  could  cost  the  Department  of 
Defense  about  $36.2  billion  by  the  year  1992. 
Software  costs  are  rising  and  will  ccMuinue  to 
rise  at  a  proportionately  greater  rate  than 
other  system  costs.  The  cost  of  software  will 
continue  to  rise  dramatically  unless  corrective 
measures  are  taken  to  include:  a  greater 
awareness  by  program  managers  about  the 
problems  faced  by  software  engineers;  an  un¬ 
derstanding  of  the  problems  that  arise  in  the 
software  development  process  and  how  they 
can  be  corrected;  and  a  change  in  the  their 
attitudes  about  software  support. 

7.2  PROBLEM  AREAS 

As  discussed  in  Chapter  2,  the  growth  in  the 
ise  of  computer  technology,  especially  in  the 
last  twenty  five  years,  has  drastically  increased 


the  requirements  for  software.  In  the  196()s 
the  Air  Force’s  F-111  aircraft  required  less 
than  100  thousand  software  instructions.  In 
the  1970s  the  Navy’s  P-3C  aircraft  required 
about  500  thousand  software  instructions.  In 
the  1980s  the  B-IB  required  over  one  million 
software  instructions  just  for  the  operational 
flight  programs.  Projections  for  the  U.S.  Air 
Forces’s  Advanced  Tactical  Fighter  (ATF) 
call  for  about  four  million  software  instruc¬ 
tions  and  the  Space  Station  calls  for  about  80 
million  software  instructions.  Software  re¬ 
quirements  for  the  next  generation  of  systems 
seem  to  be  unlimited.  These  trends  in  fielded 
software  are  depicted  in  Figure  7-1  [1]. 

This  increase  in  the  amount  of  fielded 
software  has  obviously  increased  the  require¬ 
ment  for  software  support  services.  However, 
this  increase  in  software  support  and  the  lack 
of  qualified  personnel  to  provide  these  ser¬ 
vices,  make  software  support  more  difficult  to 
manage.  Dr.  Edith  Martin  in  a  speech 
presented  in  1984  to  the  Joint  Logistics  Com- 
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Fig.  7-1  Trends  in  Software 


manders  Workshop  entitled  'The  Relation¬ 
ship  Between  PDSS  and  Advanced  Technol¬ 
ogy,"  estimated  that  the  requirement  for 
software  support  personnel  was  increasing  at 
a  rate  of  12  percent  per  year.  However,  the 
availability  of  qualified  personnel  was  only 
increasing  at  a  rate  of  4  percent.  If  one  as¬ 
sumes  an  annual  productivity  rate  gain  of  four 
percent,  there  will  exist  a  four  percent 
shortfall  in  the  amount  of  available  personnel. 
Based  on  these  estimates,  and  a.ssuming  noth¬ 
ing  else  changes,  by  1990  the  number  of  per¬ 
sonnel  required  will  exceed  the  number  of 
personnel  available  by  one  million.  An  effec¬ 
tive  way  must  be  found  to  control  both  cost 
and  schedule  and  to  properly  train,  develop, 
and  retain  a  cadre  of  software  professionals 
for  both  the  development  of  new  software  and 
the  support  of  existing  software.  Possible 
solutions  to  these  problems  will  be  discussed 
later  in  this  chapter. 


The  EIA  study  discussed  in  Chapter  2 
predicted  that  by  1992  software  will  cost  the 
DOD  about  29.1  billion  dollars,  and  computer 
hardware  will  cost  about  6.1  billion  dollars. 
Figure  7-2  indicates  that  software  support  ser¬ 
vices  will  account  for  about  seventy  percent  of 
this  anticipated  software  cost  and  software 
development  will  account  for  the  remaining 
thirty  percent.  It  is  apparent  that  if  something 
is  not  done  quickly,  the  government  will  not 
be  able  to  afford  the  software  required  by 
modern  weapon  systems. 

The  problem  of  controlling  cost  is  com¬ 
pounded  by  the  fact  that  software  support 
funding  is  fragmented.  In  1984,  the  In¬ 
dustry/Government  Workforce  Mix  Panel  of 
the  Joint  Logistics  Commanders  (JLC) 
Workshop  on  PDSS  [2]  addressed  the  Air 
Force  funding  of  system  acquisition  and 
software  support.  Even  after  a  system  is 
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FY-90  DOD  SOFTWARE  LIFE  CYCLE 

COMPUTER  BUDGET  COST 


SOFTWARE 

SUPPORT 


_ 70% 

Fig.  7-2  Software  Costs 


deployed,  hardware  and  software  are 
budgeted,  funded,  and  prioritized  by  separate 
processes  and  channels.  Multiple  procedures 
for  budgeting  and  funding  often  are  required 
for  the  same  item,  as  opposed  to  separate 
budget  and  funding  codes  for  separate  but 
related  items.  All  of  this  creates  confusion  for 
the  program  manager  or  anyone  else  trying  to 
get  a  handle  on  life  cycle  costs.  Actual  cost 
tracking  can  be  difficult  to  deal  with  and  re¬ 
quires  careful  coordination  of  one-year  R&D 
software  money  with  three-year  hardware 
procurement  money  for  system  modifica¬ 
tions.  Further  difficulties  can  develop  when 
one  considers  that  PDSS  may  be  divided  be¬ 
tween  depot  and  field-level  activities. 

The  Air  Force  has  problems  in  clearly  defin¬ 
ing  the  responsibilities  of  the  depot  and  those 
of  the  field  activities.  This  not  only  makes  it 
difficult  to  split  up  the  workload,  but  adds  to 
the  funding  problem.  The  other  services  have 
similar  problems  that  make  the  funding 
process  difficult.  If  the  government  is  to  get  a 
handle  on  cost,  the  method  used  to  fund  the 
development  process  and  PDSS  must  be 
streamlined.  There  should  be  a  common 
budgeting  procedure  used  by  all  of  the  ser¬ 


vices  and  by  all  levels  within  the  government. 
These  procedures  should  identify  appropria¬ 
tions,  budget  programs,  program  elements, 
and  specific  funding  codes  for  weapon  sys¬ 
tems  using  a  single,  simplified  process  for  set¬ 
ting  priorities.  If  this  is  accomplished,  the 
government  and  program  managers  may  at 
least  get  a  better  picture  of  the  funding  issues 
while  affording  them  a  way  to  deal  with  it  in 
an  intelligent  manner.  The  way  to  deal  with 
the  current  situation  is  to  become  aware  of  the 
difficulties  and  to  plan  in  advance  the  transi¬ 
tion  of  software  support.  This  is  done  by 
clearly  defining  the  roles  and  responsibilities 
of  the  various  support  organizations,  includ¬ 
ing  their  funding  requirements,  in  the  Com¬ 
puter  Resources  Life  Cycle  Management 
Plan  (See  Section  7-7). 

13  MANAGEMENT  PERCEPTIONS 

One  of  the  realities  of  life  is  that  people  have 
perceptions  about  software  which  may  not  be 
true.  Most  people  understand  that  hardware 
is  tangible,  material  intensive  and  produced 
by  machine,  while  software  is  intangible,  labor 
intensive,  and  mostly  produced  by  hand.  They 
further  accept  that  hardware  deteriorates 
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over  time,  is  hard  to  change,  requires  preven-  software  products.  The  software  iceberg 

live  maintenance,  and  has  relatively  high  (Figure  7-3)  is  a  graphic  description  of  some 

production  costs.  Software  is  believed  to  have  of  the  various  tools,  documentation,  etc.,  that 

exactly  the  opposite  characteristics.  Software  form  the  total  software  package.  For  the 

does  improve  over  time  with  updates,  re-  PDSSfacility  to  properly  support  the  product, 

quires  no  preventive  maintenance,  and  has  it  must  have  the  same  basic  tools  and  informa- 

only  trivial  production  costs,  but  is  never  easy  tion  that  was  available  to  the  developer.  One 

to  change  correctly.  These  perceptions  tend  should  also  realize  that  many  of  these  items 

to  cause  management  problems  and  must  be  can  have  their  own  icebergs.  There  is  a  ten- 

resolved  to  decrease  PDSS  costs.  dent7  to  cut  costs  in  software  programs  by 

cutting  out  those  things  that  are  required  later 
Tliere  is  a  mistaken  belief  that  PDSS  is  a  on  in  the  program’s  life  cycle.  Some  program 
trivial  task  and  less  important  than  software  managers  do  not  fully  understand  the  value  of 
design.  Unfortunately,  this  results  in  less  procuring  items  such  as  simulators,  editors, 

qualified  personnel  assigned  to  do  software  compilers,  test  tools,  other  software  tools,  and 

support  rather  than  the  more  skilled  tech-  documentation.  Some  may  even  feel  that 

nicians  and  managers.  The  feeling  is  that  such  items  are  gold  plating.  The  fact  is  that 

software  support  personnel  should  po.ssess  without  these  items  software  cannot  be 

the  same  skill  levels  as  the  local  auto  properly  supported, 

mechanic  while  software  design  personnel 

should  have  the  skills  of  automotive  en-  7.4  MANAGEMENT  CONCERNS 
gineers. 

Program  Managers  must  understand  the 
Another  misconception  that  plagues  some  needs  of  the  user.  They  are  often  faced  with 

program  managers  is  a  lack  of  understanding  the  need  to  change  the  software  because  of  a 

as  to  what  constitutes  the  supportable  new  requirement  (e.g.,  a  changing  threat)  or 


Fig.  7-3  The  Software  iceburg 
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to  correct  a  problem.  All  program  managers 
should  ensure  that  the  software  developed  is 
supportable,  and  that  the  capability  to  support 
the  software  by  the  government  or  industry 
exists.  This  determination  of  support  must 
include  the  people  who  will  actually  do  the 
work  as  well  as  the  facilities  and  tools  neces¬ 
sary  to  accomplish  this  effort.  Cost  and 
schedule  are  obvious  drivers  in  ilie  decision 
process.  However,  software  considerations 
must  be  put  in  perspective  with  the  total 
weapon  system.  Any  change  to  the  software 
may  have  an  impact  on  the  hardware.  In  ad¬ 
dition,  any  changes  made  to  the  software  may 
impact  other  software  and/or  future  system 
requirements,  just  as  hardware  changes  have 
impact  on  the  system. 

7.5  WHATISPDSS? 

Software  does  not  fail  in  the  classical  sense. 
Hardware  degrades  over  time  as  components 
wear  out.  A  software  problem  is  due  to  an 
error  that  has  existed  since  its  creation. 
When  a  problem  caused  by  a  component 
failure  is  found  in  hardware,  the  solution  en¬ 
tails  bringing  the  item  back  to  the  original 
configuration.  In  software  when  a  problem  is 
found  and  corrected,  a  new  configuration  is 
created.  Software  does  not  wear  out  like 
hardware;  so  "software  maintenance"  is  a  mis¬ 
nomer.  The  appropriate  name  for  this  effort 
is  "software  support".  The  Joint  Lx)gistics 
Commanders  in  1984  decided  that,  in  order  to 
answer  the  many  questions  about  software 
support,  a  definition  of  Post  Deployment 
Software  Support  was  needed.  Inirther,  such 
a  definition  should  provide  a  uniform  basis  for 
understanding  and  dealing  with  software  sup¬ 
port  issues.  The  JLCs  have  defined  PDSS  as 
follows: 

"Post  Deployment  Software  Support  is  the  sum 
of  all  activities  required  to  ensure  that,  during 
the  production/deployment  phase  of  a  mission 


critical  computer  system ’s  life,  the  implemented 
and  fielded  softwarelsystem  continues  to  sup- 
pon  its  original  operational  mission  and  sub¬ 
sequent  mission  modifications  and  production 
improvement  efforts." 

This  means  that  software  is  modified  to  either 
correct  a  problem  or  to  add  a  capability.  There 
are  other  ways  to  look  at  the  various  software 
support  activities  other  than  breaking  down 
the  efforts  into  modification  and  the  addition 
of  capabilities.  Figure  7-4  [3]  looks  at  two 
different  approaches.  Swanson  [4]  breaks 
down  software  support  into  three  categories. 
The  first  category  talks  about  corrective  ef¬ 
forts,  examples  of  which  might  be  the  correc¬ 
tion  of  a  problem  in  reading  a  file  in  a  record 
properly,  or  a  failure  to  test  all  possible  con¬ 
ditions.  The  second  category  deals  with  adap¬ 
tive  efforts  which  include  such  things  as 
improving  processing  speed  or  adjustments  to 
add  new  input  or  output  devices.  The  last 
category  includes  software  that  is  modified  to 
make  enhancements  in  response  to  new  re¬ 
quirement, s,  or  to  give  the  operator  more 
flexibility.  Reutter  [5]  refined  these  efforts  by 
breaking  them  into  the  seven  categories 
shown  in  Figure  7-4.  Reutter’s  seven 
categories  separate  support  from  each  of 
Swanson’s  three  categories.  'Phe  intention  of 
this  breakout  is  to  emphasize  the  communica¬ 
tion  between  the  user  and  the  support  activity. 
It  akso  shows  the  importance  of  planning  for 
support. 

There  is  some  disagreement  among  software 
people  as  to  whether  the  above  efforts  should 
be  called  software  support  or  software  main¬ 
tenance.  The  term  software  maintenance 
gives  the  impression  that  the  effort  involves 
preventing  components  from  wearing  out  or 
breaking  as  in  the  case  of  hardware.  This  is 
not  the  case  for  software  and  one  of  the 
reasons  why  the  JLCs  use  the  term  software 
support.  Also  the  term  maintenance  does  not 
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Swanson’s  Categories 

Reutter’s  Categories 

CORRECTIVE  MAINTENANCE 

- - 

- . 

EMERGENCY  REPAIRS 

Performed  to  Identify  and  correct 

Performed  when  immediate  repair  is 

software  failures,  performance 

necessary  to  continue  user  service. 

failures,  and  implementation 

failures. 

CORRECTIVE  CODING 

Performed  to  correctly  reflect  the 

specification  or  to  correctly  utilize 

system  resources. 

ADAPTIVE  MAINTENANCE 

. 

UPGRADES 

Performed  to  adapt  software  to 

'x 

Performed  to  adapt  to  changes 

changes  in  (he  data  requirements 

in  processing  requirements. 

or  the  processing  environments. 

CHANGES  IN  CONDITIONS 

Performed  to  adapt  to  changes 

in  business  conditions  due  to 

regulatory  situations  or  other 

situations  beyond  the  control 

of  the  organization. 

PERFECTIVE  MAINTENANCE 

Performed  to  '.nhance  perform¬ 
ance,  improve  cost-effectiveness 

improve  processing  efficiency. 

GROWTH 

Performed  to  adapt  to  changes  in 

data  requirements  or  the  addition 

of  new  programs,  or  new  users. 

or  improve  maintainability. 

ENHANCEMENTS 

Performed  in  response  to  users 

requests  for  changes  and  additions 

to  the  system. 

SUPPORT 

Performed  to  explain  system 

capabilitie:..  to  plan  for  future 

support,  to  measure  performance. 

Fig.  7-4  Types  of  Software  Support  EITorts 


convey  a  feeling  that  design  work  is  being 
carried  out.  In  any  case,  the  amount  of  work 
required  to  correct  software  deficiencies  ac¬ 
counts  for  only  about  twenty  percent  of  tiie 
total  software  support  effort  [b].  Eighty  per¬ 
cent  of  the  PDSS  efforts  are  spent  in  adding 
new  capability  to  the  software,  or  refining 
recjuircments  not  clearly  defined  in  the 
o  r  i  g  i  n  a  1  r  e  t|  u  i  r  e  n  I  e  n  t  s . 


7.6  SOFTWARE  LIFE  CYCLE  CON- 
SI  DERM  IONS 

One  of  the  major  reasons  for  the  high  cost  of 
software  support  is  that  it  costs  much  more  to 
correct  problems  after  the  initial  design  has 
been  completed  than  it  does  to  correct 
problems  early  in  the  devek  oment  phase. 
Modifications  require  the  same  development 
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activities  as  the  original  process  because 
software  support  is  redevelopment.  It  is  ex¬ 
tremely  difficult  to  understand  someone 
else’s  design,  to  figure  out  what  they  were 
trying  to  accomplish,  to  correct  any  latent 
problems,  or  to  add  new  capabilities.  This  can 
become  impossible  if  the  documentation  is 
poor. 

Figure  7-5  indicates  that  while  most  errors  are 
introduced  during  the  early  phases  of  the 
software  development  cycle,  these  errors  are 
not  usually  found  until  the  software  is  being 
validated  or  supported.  Barry  W.  Boehm  [  IJ 
indicates  that  the  cost  of  correcting  software 
errors  or  adding  lines  of  code  to  existing 
software  inereases  dramatically  as  the  life 
cycle  progresses.  'Fherc  ctni  be  as  much  as  a 
hundred  to  one,  or  greater,  incretise  in  costs. 
In  a  study  performed  in  1978  [7J,  the  DOD 
found  that  the  average  cost  of  generating  a 


late  in  the  development  cycle,  tends  to  sig¬ 
nificantly  increase  software  life  cycle  costs. 

7.7  IMPROVING  THE  PDSS  PROCESS 

There  are  a  number  of  actions  that  can  be 
taken  to  improve  the  PDSS  process  and  to 
reduce  software  life  cycle  costs.  One  of  the 
most  important  is  the  creation  of  a  document 
that  spells  out  all  of  the  activities  that  must  be 
accomplished  during  the  life  of  the  software 
program.  Within  the  DOD  this  document  is 
called  the  Computer  Resources  Life  Cycle 
Management  Plan  (CRLCMP).  'Phe 
CRLCMP  is  developed  early  in  the  acquisi¬ 
tion  cycle  to  ensure  that  all  issues  tind  resour¬ 
ces  relevant  to  the  acc]uisition,  testing,  and 
support  are  properly  accounted  for.  Both  the 
Navy,  with  OPNAVINS'I’ 5200.28  [8]  and  the 
Air  Force  with  AFR  800-14  [9]  rct|uirc  that 
the  CRLCMP  be  inititited  by  the  developing 
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line  of  code  is  about  seventy-five  dollars  while 
the  average  cost  of  modifying  a  line  of  code 
late  in  the  development  cycle  or  after  software 
delivery  is  four-thousand  dollars.  This  high 
cost  of  modifying  software,  coupled  with  the 
fact  that  most  errors  are  not  uncovered  until 


activity  during  Concept  Lixploration  ;md  u[)- 
dated  as  the  program  progresses,  d’he 
CRI.C'MP  is  approved  prior  to  Full  Scale 
Develo[)ment.  However,  it  should  be  clearly 
understood  that  the  CRLCMP  is  a  living 
document  and  should  be  u|)dated  whenever 
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the  software  is  modified.  A  rule  of  thumb  is  to 
update  the  CRLCMP  at  least  annually. 

The  CRLCMP  describes  the  total  software 
support  strategy.  It  defines  the  criteria  for 
measuring  progress  and  identifies  the  resour¬ 
ces  needed  to  develop,  test,  acquire,  and  sup¬ 
port  computer  resources  (e.g.,  facilities, 
personnel,  hardware,  software,  training,  fund¬ 
ing,  tools).  The  CRLCMP  identifies  the 
regulations  and  operating  instructions  that 
will  be  used  to  manage  the  system  software.  It 
also  identifies  all  the  organizations  involved 
in  the  acquisition  and  support  and  their  roles, 
responsibilities  and  relationships.  The  In¬ 
tegrated  Lx)gistics  Support  Plan  (ILSP)  is  the 
[)arent  document  to  the  CRLCMP  and 
defines  the  overall  su[)portability  strategy. 
Therefore,  the  CRLCMP  should  be  clo.sely 
coordinated  with  the  ILSP. 

Managing  PDSS  is  managing  change.  It  is 
important  to  put  a  rigorous  change  control 
process  in  place  during  development  that  can 
effectively  transition.  The  program  manager 
and/or  support  manager  must  understand  why 
software  changes  are  needed  and  what  resour¬ 
ces  are  needed  to  make  appropriate  economic 
and  technical  tradeoffs.  An  effective  change 
control  system  allows  the  program  mtmager  to 
make  these  decisions. 

PDSS  managers  must  understand  their  role  in 
the  PDSS  process  and  the  motivation  for  a 
software  change.  Is  a  change  required  to  add 
a  capability  that  the  user  needs,  or  to  correct 
a  deficiency?  If  a  change  is  required,  are  the 
personnel  necessary  to  implement  the  change 
available  and  capable  of  performing  the  task? 
1  las  the  program  office  provided  the  software 
sup[)ort  personnel  the  necessary  tools  and 
resources  nccessai^  to  mtike  changes  success¬ 
fully?  When  a  change  is  to  be  made,  will  there 
be  any  disruption  to  current  services?  I  low 
long  will  it  take  to  make  these  changes?  What 


will  be  the  impact  on  software  integrity  and 
what  are  the  ramifications  for  future  changes? 

In  1984  the  JLC  report  on  the  "Cost  of  Owner¬ 
ship"  [2]  concluded  that  program  managers 
needed  to  understand  how  each  PDSS  activity 
was  organized  and  how  it  functions  within 
their  own  services.  To  this  end,  they  provided 
the  following  descriptions  of  PDSS  centers: 

Army  -  The  Army  PDSS  center  is  a  center 
within  a  DARCOM  subordinate  command 
established  to  support  the  software  subsys¬ 
tems  of  all  battlefield  automated  .systems  for 
v/hich  that  command  has  logistics  support 
responsibility.  Each  center  normally  sup¬ 
ports  numerous  .systems. 

Navy  -  The  Navy  PDSS  centers’  functions  and 
staffing  are  provided  for  by  the  In-Service 
Engineering  Activity  assigned  the  life  cycle 
.system  support  responsibility.  Note  that  a  .sys¬ 
tem  may  be  an  aircraft  avionics  packiige,  a 
shipboard  navigational  system,  or  a  shore- 
based  Command,  Control,  Communications 
and  Intelligence  system. 

Air  F<'orce-The  Air  Force  provides  fora  PDSS 
center  as  jnirt  of  Integrated  Support  I-acility 
(ISF)  which  is  used  to  provide  all  hardware 
and  software  engineering  support.  This  ISF’ is 
located  in  the  engineering  division  or  branch 
which  supports  the  .system  program  director 
(SPD)  in  an  Air  L)gi.siics  Center. 

Marine  Corps  -  The  Marine  Corps  has  estab¬ 
lished  a  single  PDSS  center  completely 
separate  from  hardware  maintenance 
facilities.  This  center  provides  su[)port  for 
designated  Marine  Corps  Softwtire  programs. 

1.  Organization  Chain; 

Within  the  services,  PDSS  centers  are  located 
either  in  a  logistics  chain,  a  Research  and 
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Development  (R&D)  chain  or  a  combination 
of  the  two.  The  Navy  has  a  combination  chain 
with  a  single  boss.  The  Air  Force  PDSS  cen¬ 
ter  is  in  the  logistics  chain,  bnt  receives  direc¬ 
tion  from  a  logistics  boss  via  the  R&D  chain. 
The  Army  established  If  PDSS  centers  lo¬ 
cated  at  the  development  commands,  but 
funded  by  the  readiness  organizations  within 
combined  commands.  Overall  PDSS 
management  is  performed  by  DARCOM. 

Coordination  between  R&D  and  logistics  is 
always  difficult.  Maving  a  single  boss  reduces 
the  difficulty  to  some  degree. 

2.  Devclupnieiit  of  Policy  and  Compliances; 

Higher  level  headquarters  esttihlishes  policy, 
publishes  implementing  instruction  and  en¬ 
sures  compliance  by  the  PDSS  centers  within 
their  individual  commtinds. 

3. 1  low  Putuled: 

The  Air  Force  is  0[)erations  and  Main¬ 
tenance  (O&M)  funded  unless  a  nuijor 
rebuild  is  required;  then  the  system  goes  back 
to  the  developer  tmd  R&D  funds  tire  used. 
The  Navy  primttrily  uses  O&M  funds,  hut  may 
also  send  major  motlifictUions  back  into  a 
R&D  cycle,  'i’he  Marine  Corps  uses  R&D 
funds.  'I'he  Army  uses  a  plethora  of  funding 
including  numerous  types  of  O&M,  procure¬ 
ment  and  R&D  dollars. 

A  standard  approach  to  funding  and  a  better 
definition  of  maintenance  would  help  reduce 
some  of  these  overly  burdensome  reciuisition 
tmd  accounting  functions. 

4.  Acquire  Software  F-nvironment: 

In  all  the  services,  the  PDSS  centers,  in  con¬ 
junction  with  the  tieveloper,  identify  support 
requirements.  In  the  Ntivy  and  Air  I’orce  the 


acquisition  manager  is  responsible  for  procur¬ 
ing  the  initial  suites  of  equipments,  and  the 
PDSS  center  is  responsible  for  updat¬ 
ing/replacing  that  equipment.  In  the  Army, 
no  defined  responsibility  exists  to  ensure  that 
the  developer  acquires  the  support  environ¬ 
ment,  including  mockups  and  simulators. 

5.  How  Location  Is  Determined; 

The  Air  Force  locates  the  PDSS  centers 
within  the  system  program  directorate  (Air 
D)gistics  Center)  along  with  sustaining  en¬ 
gineering.  The  Navy  collocates  the  PDSS 
centers  with  the  activity  responsible  for  in- 
service  engineering  support.  The  Marine 
Corps  only  has  one  PDSS  center  whose  com¬ 
mand  has  the  logistics  responsibility  for  the 
system  or  has  computer  resources.  If  the  sys¬ 
tem  is  a  command  and  control  system,  the 
PDSS  center  is  collocated  with  the  battlefield 
functional  area  school. 

6.  How  System  is  I  . earned; 

The  Army  and  Air  I’orce  PDSS  centers  be¬ 
come  involved  at  the  beginning  of  the 
development  cycle.  'I’hey  either  [)articip;itc  in 
the  developmcnttil  process  or  become  the 
IV&  V  orgtmization.  The  Navy  mtiy  follow  the 
same  [)roccdure,  v!e[K'nding  on  when  the 
PDSS  center  is  designated.  The  Marine 
C'orps  PDSS  center  has  pieviously  been  in¬ 
volved  as  part  of  the  development  respon¬ 
sibility  and  has  replaced  it  with  more  of  an 
IV&V  type  role. 

I'he  involvement  of  the  PDSS  centers 
throughout  the  development  cycle  is  critical 
to  the  successful  performance  of  PDSS  work. 

7.  Use  Of  the  PDSS  Tenter  For  IV&V: 

d'here  is  currently  no  stated  rec]uirement  to 
perform  IV&V  in  any  of  the  services,  and 
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there  is  a  wide  variance  of  how  the  services 
accomplish  IV&V.  In  those  programs  where 
there  is  a  requirement  for  IV&V,  the  PDSS 
center  is  a  logical  choice  and  should  be  used 
to  the  maximum  extent  practical. 

8.  Software  Configuration  Control; 

All  service  PDSS  centers  perform  configura¬ 
tion  control,  but  are  not  always  formally 
tasked  with  performing  configuration 
management. 

9.  Type  of  Changes: 

I'here  are  three  types  of  changes:  those 
brought  about  by  latent  defects;  those  brought 
about  by  user  enhancement  requests;  and 
those  necessitated  by  major  product  improve¬ 
ments.  All  the  service  PDSS  centers  perform 
the  first  two  types  of  changes.  Major  product 
improvements  are  usually  accomplished  by  a 
contractor,  with  the  PDSS  center  providing 
btickground  information  and  support. 

10. 1‘Aaluaf  ion  Of  Complaint: 

The  Army  mtiintenance  directorate  sends 
logistic  support  repre.scntalives  to  the  user 
activity  to  investigate  com[)laints.  Once  the 
[)roblem  is  identified  and  verified,  the  main¬ 
tenance  directorate  notifies  the  PDSS  centers 
who  then  attempt  to  duplicate  the  problem. 
The  other  service  PDSS  centers  receive  the 
trouble  report  directly  from  the  user  and  at¬ 
tempt  to  duplicate  the  prob.'em.  Responses 
btick  to  the  com|)laining  user  vary  from  peri¬ 
odic  stiitus  reports  about  the  complaint  to  not 
providing  any  follow-up  information. 

II.  Develop  Software  Engineering  Change 
Solution; 

Once  the  problem  has  been  identified,  the 
PDSS  centers  determine  the  cause.  Solu¬ 


tions  are  developed  and  testing  is  conducted 
to  ensure  that  the  original  problem  has  been 
solved  and  that  additional  problems  have  not 
been  created. 

12.  Integration  Testing; 

In  all  the  services,  integration  testing  is  per¬ 
formed  when  the  PDSS  center  has  completed 
system  testing.  With  the  exception  of  the 
Navy,  testing  is  always  performed  on  the  ac¬ 
tual  equipment  being  integrated.  In  the  Navy, 
the  size  of  the  integration  problem  often 
prevents  the  PDSS  center  from  conducting 
the  integration  testing  in  a  totally  realistic 
environment.  The  software  which  has  been 
modified  is  run  on  actual  system  hardware, 
but  those  systems  with  which  it  communicates 
(i.e.,  integrated)  may  be  simulated.  The 
limitations  on  integration  testing  of  Navy 
shipboard  combat  systems  due  to  the 
availability  of  actual  systems  are  recognized 
and  organic  integration  facilities  have  been  or 
are  being  established.  These  facilities  are 
outside  the  PDSS  centers’  responsibility  and 
control,  but  are  available  to  the  PDSS  center 
for  use. 

13.  Interoperability  Testing: 

All  the  service  PDSS  centers  conduct  inter¬ 
operability  testing  to  ensure  that  changes 
made  to  correct  problems  will  in  no  way  inter¬ 
fere  with  the  capabilities  of  the  .system  to 
communicate  with  other  systems. 

14.  Documentation  Update: 

All  the  service  PDSS  centers  update 
documentation  for  every  change  made.  Two 
major  problems  being  experienced  are  the 
inadequacy  of  many  existing  standards  and 
initial  delivery  of  poor  documentation. 
Standards  must  be  published  that  meet  the 
needs  of  all  services  and  contracts  must  be 
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written  to  require  documentation  in  accord¬ 
ance  with  these  standards.  DOD-STD-2167A 
is  a  positive  step  in  the  right  direction.  It 
describes  a  common  approach  to  software 
development. 

15.  Distribution  Of  Software  Corrections  To 
Users: 

For  systems  where  there  is  limited  distribu¬ 
tion,  changes  are  hand-delivered  and  accom¬ 
panied  by  sufficient  instruction  to  allow  the 
user  to  execute  a  smooth  transition.  In  those 
systems  where  a  large  number  of  changes 
must  be  installed,  the  users  are  supplied  with 
a  written  instruction  package  through  a  dis¬ 
tribution  process. 

7.8  MANAGEMENT  GUIDANCE. 

By  changing  the  traditional  prejudices  toward 
software  support,  by  following  sound  en¬ 
gineering  practices,  and  through  the  use  of 
good  management  tools  and  plans,  program 
managers  can  begin  to  deal  with  the  software 
life  cycle  support  problems.  The  following 
represents  a  set  of  guidelines  which  can  be 
followed  by  the  program  manager  and  can 
lead  to  more  cost  effective  software  support: 

(a)  The  program  manager  must  ensure  that 
sound  software  engineering  design  techni¬ 
ques  are  used  during  the  development 
process  where  the  majority  of  all  errors  are 
introduced  and  that  the  software  is  designed 
for  supportability; 

(b)  The  CRLCMP  must  be  developed  early  in 
the  life  of  the  program  development  cycle 
(Concept  Exploration  phase),  periodically 
updated  and  adhered  to; 

(c)  Personnel  and  productivity  are  important 
issues.  It  is  incumbent  upon  program 
managers  to  acquire  and  retain  a  qualified  and 


stable  workforce.  This  may  require  the  pro¬ 
gram  manager  to  develop  a  continuous  train¬ 
ing  program  for  software  personnel.  In  order 
to  get  more  out  of  the  personnel,  program 
managers  must  provide  the  supporting  ac¬ 
tivity  modern  tools  and  facilities; 

(d)  Documentation  and  support  software 
tend  to  be  cut  out  of  programs  early  in  the 
development  process  in  order  to  save  money. 
This  causes  severe  problems  later  on  in  the 
program.  It  is  important  that  all  the  tools 
necessary  to  provide  software  support  are 
delivered  to  the  support  facility; 

(e)  The  software  baseline  must  be  controlled 
throughout  its  life  cycle  by  using  good  con¬ 
figuration  management  techniques; 

(f)  The  program  manager  should  involve  the 
PDSS  organization  early  in  the  development. 
The  personnel  at  these  facilities  have  a  wealth 
of  knowledge  about  what  is  required  to  sup¬ 
port  a  program  and  about  the  associated 
problems.  This  provides  the  program 
manager  with  a  real  time  lessons  learned; 

(g)  It  is  imperative  that  the  program  manager 
appropriately  plan,  budgets,  and  fund  the 
PDSS  effort. 
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CHAPTER  8 


PLANNING  FOR  COMPUTER  SOFTWARE 


8.1  INTRODUCTION 

This  chapter  will  discuss  the  software  plan¬ 
ning  activities  that  must  be  conducted  during 
tne  various  phases  of  software  acquisition. 
The  success  of  any  project  is  greatly  deter¬ 
mined  by  how  much  care  and  time  are  put  into 
the  planning  process.  Software  development 
and  acquisition  are  no  different! 

Planning  for  computer  software  begins  during 
the  Concept  Exploration  (CE)  Phase  and  in¬ 
tensifies  during  the  Demonstration  and 
Validation  (D/V)  Phase.  By  the  time  the  pro¬ 
gram  enters  the  Full  Scale  Development 
(FSD)  Phase,  most  of  the  necessary  software 
planning  documents  should  have  been 
generated. 

8.2  PLANS  AND  DOCUMENTATION 

The  major  plans  and  documents  generated  by 
the  program  office  are  the  Program  Manage¬ 
ment  Plan  (PMP),  the  Test  and  Evaluation 
Master  Plan  (TEMP),  the  Integrated  la)gis- 


tics  Support  Plan  (ILSP),  the  Computer 
Resources  Lifecycle  Management  Plan 
(CRLCMP),and  the  System/Segment 
Specification  (SSS). 

8.2.1  Program  Management  Plan  (PMP) 

The  purpose  of  the  PMP  is  to  guide  all  pro¬ 
gram  office  personnel  toward  a  common  goal. 
The  PMP  provides  essential  information  on 
the  overall  program  strategy  and  goals.  It  can 
be  viewed  as  the  game  plan  for  the  program 
office  and  includes  a  schedule  of  the  major 
events  leading  to  the  initial  operating 
capability  (IOC). 

The  PMP  should  be  updated  and  kept  current, 
particularly  with  respect  to  the  schedules.  It 
should  be  mandatory  reading  for  every  new¬ 
comer  into  the  organization.  This  is  especially 
true  in  software  development  since  software 
is  usually  on  the  critical  path.  A  detailed  out¬ 
line  of  the  PMP  is  given  in  Appendix  C. 
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8.2.2  Test  and  Evaluation  Master  Plan 
(TEMP) 

The  TEMP  is  the  basic  planning  document  for 
all  test  and  evaluation  (T&E)  related  to  a 
particular  system  acquisition.  It  is  used  by  the 
Office  of  the  Secretary  of  Defense  (OSD)  and 
all  DOD  components  in  planning,  reviewing, 
and  approving  T&E.  The  TEMP  provides  the 
basis  for  all  other  detailed  T&E  planning 
documents. 

The  TEMP  must  address  the  effects  of  human 
performance  on  the  weapon  system’s  opera¬ 
tional  effectiveness  and  the  ability  of  the  sys¬ 
tem  to  meet  performance  standards, 
including  reliability  and  maintainability. 

I'he  TEMP  must  address  the  system’s  critical 
technical  performance  thresholds  and  their 
relationship  to  the  system’s  required  opera¬ 
tional  characteristics.  It  must  clearly  outline 
the  planned  T&E  process  through  which  the 
test  objectives  will  be  met.  It  must  also 
describe  the  physical  hardware  tests  and  any 
analysis  required  to  provide  data  not  gained 
from  actual  testing. 

The  TEMP  must  clearly  describe  the  required 
T&E  activities  along  with  all  the  necessary 
resources  for  fully  testing  the  operational 
suitability  of  the  weapon  system. 

The  TEMP  is  intended  to  be  a  living  docu¬ 
ment  that  addresses  the  changing  critical  is¬ 
sues  affecting  any  acquisition  program. 
Major  changes  in  program  requirements, 
schedule,  or  funding  usually  result  in  a  change 
to  the  test  program.  To  ensure  that  T&u. 
requirements  are  current,  the  TEMP  shall  be 
updated  on  an  annual  basis  until  all  significant 
testing  is  complete.  The  update  shall  reflect 
changes  to  the  T&E  program  due  to:  test 
results,  changes  in  the  scope  or  schedule  of 
the  T&E  program,  changes  in  the  required 


characteristics,  changes  in  the  reassessment 
of  test  resource  provisions  and  limitations,  or 
any  changes  deemed  necessary  by  the  OSD. 
A  detailed  outline  of  the  TEMP  is  found  in 
Appendix  D. 

8.2.3  Integrated  Logistics  Support  Plan 
(ILSP) 

The  ILSP  describes  and  documents  the  in¬ 
tegrated  logistics  support  program.  It  is  the 
principal  logistics  document  for  an  acquisi¬ 
tion  p»'ogram  and  serves  as  a  source  d  '  ;ument 
for  summary  and  consolidated  information 
required  in  other  program  management 
documents.  It  is  summarized  in  the  System 
Concept  Paper  (SCP)  and  the  Decision  Coor- 
dinaMon  Paper  (DCP). 

The  purpose  of  the  ILSP  is  to: 

(a)  Provide  a  complete  plan  for  support  of  the 
fielded  system; 

(b)  Provide  details  of  the  ILS  program  and  its 
relationship  with  overall  program  manage¬ 
ment; 

(c)  Provide  decision  making  bodies  with  the 
necessary  information  on  ILS  aspects  for 
making  sound  decisions  on  further  develop¬ 
ment  and  production  of  the  basic  system; 

(d)  Provide  the  basis  for  the  preparation  of 
the  ILS  sections  of  the  procurement  package, 
e.g.,  statement  of  work  (SOW),  specification, 
and  source  selection  and  evaluation  criteria; 

(e)  Describe  how  readiness  and  sustainability 
will  be  achieved. 

The  ILSP  is  initially  a  section  of  the  Program 
Management  Plan  (PMP)  but,  early  in  the 
Demonstration  and  Validation  Phase,  it  is 
removed  from  the  PMP  and  becomes  a  stand- 
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alone  document.  At  a  minimum,  the  ILSP  is 
updated  annually.  A  detailed  outline  of  the 
ILSP  is  found  in  Appendix  E. 

8.2.4  Computer  Resources  Life  Cycle 
Management  Plan  (CRLCMP) 

The  CRLCMP  is  the  primary  planning  docu¬ 
ment  for  computer  resources  throughout  the 
system  life  cycle.  It  complements  the  In¬ 
tegrated  Logistics  Support  Plan.  The  purpose 
of  the  CRLCMP  is  to: 

(a)  Document  the  software  support  concept 
and  the  resources  needed  to  achieve  the  sup¬ 
port  posture; 

(b)  Document  the  computer  resources 
development  strategy; 

(c)  Identify  the  applicable  directives  (regula¬ 
tions,  operating  instructions,  technical  orders, 
etc.)  for  managing  computer  resources  in  the 
system; 

(d)  Define  any  changes  or  new  directives 
needed  for  the  operation  or  support  of  com¬ 
puter  resources; 

(e)  Define  the  scope  of  independent  verifica¬ 
tion  and  validation  (IV&V)  efforts. 

Development  of  the  CRLCMP  is  initiated 
during  the  Concept  Exploration  phase.  The 
CRLCMP  is  coordinated  with  the  user  and 
supporting  organizations  before  release  c 
the  Full  Scale  Development  solicitation. 

During  the  Production  and  Deployment 
phases,  the  CRLCMP  is  updated,  as  required, 
to  reflect  significant  changes  in  the  system  or 
its  support  environment.  When  updating  the 
CRLCMP,  sections  that  refer  to  ac¬ 
complished  events  should  be  reworded  as  his¬ 
torical  notes  or  deleted.  After  the  system  is 


transitioned  to  the  user  the  software  support 
activity  will  assume  responsibility  for  the 
CRLCMP.  A  detailed  outline  of  the 
CRLCMP  is  found  in  Appendix  F. 

8J  ENGINEERING  STUDIES 

Systems  engineering  studies  are  based  on  the 
concept  of  a  hierarchy  of  requirements  start¬ 
ing  with  system  level  requirements  and  en¬ 
ding  with  detailed  engineering  specifications 
and  data.  System  definition  proceeds  by 
refining  each  level  of  requirement  into  subor¬ 
dinate  requirements  until  the  entire  system  is 
described.  Computer  resources  are  con¬ 
sidered  as  an  integral  part  of  the  system  and 
are  subject  to  tradeoff  and  optimization 
studies.  Systems  engineering  studies  will  nor¬ 
mally  include: 

Requirements  Definition  -  Requirements 
definition  begins  with  the  creation  of  the  sys¬ 
tem  level  requirements  specification.  This 
definition  may  be  spelled  out  in  either  the 
System/Segment  Specification  (SSS),  the 
Prime  Item  Development  Specification 
(PIDS),  or  the  Critical  Item  Development 
Specification  (CIDS).  The  developer  then 
performs  the  necessary  analysis  to  determine 
the  preliminary  allocation  of  the  require¬ 
ments  between  hardware  and  software.  In  ad¬ 
dition,  the  developer  begins  work  on  the 
preliminary  Software  Requirements 
Specification  (SRS)  and  the  preliminary  In¬ 
terface  Requirements  Specification  (IRS). 
The  formal  allocation  of  what  is  to  be  ac¬ 
complished  in  hardware  and  what  is  to  be 
accomplished  in  software  is  documented  in 
the  Systems/Segment  Design  Document 
(SSDD).  These  documents  are  necessary  for 
the  effective  accomplishment  of  the  software 
design.  The  System  Specification  and  the 
SSDD  are  approved  at  the  Systems  Design 
Review  (SDR)  and  a  Functional  Baseline  es¬ 
tablished. 
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Interface  Definition  -  The  Computer 
Resources  Working  Group  (CRWG)  and  the 
Interface  Control  Working  Group  address 
system  and  subsystem  interface  requirements 
that  may  affect  computer  resources.  The  re¬ 
quirements  for  these  interfaces  are  docu¬ 
mented  in  system  specifications  (i.e.,  SSS, 
PIDS,  CIDS).They  are  furthered  detailed  in 
the  IRS.  The  IRS  and  the  SRS  are  later  ap¬ 
proved  at  the  Software  Specification  Review 
and  together  these  two  documents  formally 
define  the  Allocated  Baseline. 

TradeofT  and  Optimization  -  Tradeoff  and 
optimization  studies  should  consider: 

(a)  Tradeoffs  between  computer  software  and 
computer  hardware; 

(b)  Required  computer  processor  architec¬ 
tural  features  such  as  memory  size,  speed, 
input  and  output  capacity,  and  spare  capacity; 

(c)  Use  of  standard  equipment,  HOLs,  in¬ 
struction  set  architectures,  and  interfaces; 

(d)  Alternate  approaches  for  meeting  system 
security  requirements; 

(e)  Improved  supportability  versus  improved 
performance; 

(f)  Use  of  existing  government  resources  or 
commercial  off-the-shelf  resources  versus 
new  development. 

Feasibility  Studies  -  These  studies  determine 
the  feasibility  of  alternative  allocations  of  .sys¬ 
tem  requirements  to  computer  resources  and 
the  derivation  of  data  for  formulating  budgets 
and  schedules. 

Risk  Analysis  -  Identify  the  major  software 
development  risks  using  Table  8-1  and  incor¬ 


porate  into  the  system  risk  management  plan 
or  the  CRLCMP. 

Software  Support  Studies  -  Software  support 
studies  are  conducted  to  refine  the  system 
support  concept,  to  allocate  software  support 
requirements,  and  to  identify  operational  sys¬ 
tem  software. 

8.4  THE  COMPUTER  RESOURCES 
WORKING  GROUP  (CRWG) 

A  CRWG  should  be  established  as  early  as 
possible  during  the  CE  Phase  but  no  later  than 
Milestone  I.  For  modification  programs,  and 
those  acquisitions  closely  related  to  ongoing 
programs,  an  existing  CRWG  may  be  used. 
The  CRWG’s  role  is  to  participate,  in  an  ad¬ 
visory  capacity,  in  all  computer  resources 
aspects  of  the  program.  This  includes  pro¬ 
gram  management  review.s,  source  selection 
evaluation  boards,  design  reviews,  and  audits. 

The  CRWG  is  formally  chartered  by  the  pro¬ 
gram  manager  and  should  coordinate  its  ac¬ 
tivities  with  the  operational  user,  the 
supporting  organization,  the  interface  control 
working  group,  and  any  other  organization 
with  an  active  interest  in  the  program.  At  a 
minimum  the  CRWG  will: 

(a)  Advise  the  program  manager  in  all  areas 
relating  to  the  acquisition  and  support  of  com¬ 
puter  resources; 

(b)  Generate  the  initial  CRLCMP  and  update 
it  as  the  program  progresses; 

(c)  Select  a  software  support  concept  and 
document  it  in  the  CRLCMP; 

(d)  Monitor  compliance  of  the  program  with 
computer  resources  policy,  plans,  procedures, 
and  standards; 
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CAUSE 

ACTION 

Lack  of  adequate  definition  of  computer  resource  functional, 
interface,  support,  or  performance  requirements  prior  to 
structuring  the  program. 

Poorly  defined,  complex,  or  untestable  Intra-  or  inter-system 
interfaces,  including  human  interfaces. 

Lack  of  stability  in  computer  resource  requirements  during 
development. 

System  engineering  techniques  such  as  funtional  analyses, 
simulation,  mathematical  modeling,  correctness  proofs,  and 
tradeoff  analyses. 

Incremental  development  strategies  which  tackle  large,  com¬ 
plex,  and  poorly  understood  requirements  in  smaller,  more 
manageable  parts. 

Lack  of  government  visibility  into  the  contractor's  software 
development  effort. 

Rigorous  application  of  traditional  cost,  schedule,  and  per¬ 
formance  tracking  techniques  with  careful  attention  to 
earned  value  progress  against  measureable  milestones. 

Since  these  techniques  are  almost  always  driven  by  the 

W8S,  visibility  of  critical  and  high  risk  computer  resources 
is  a  primary  criterion  for  determining  the  appropriate  level 
within  the  WBS  for  these  components  of  the  system. 

Use  of  a  risk  tracking  system  to  collect  data  on  the  status 
of  identified  high  risk  items.  The  output  of  this  system 
should  be  a  standard  part  of  periodic  reviews. 

Use  of  independent  verification  and  validation. 

Performance  requirements  that  push  the  state  of  the  art. 

Prototyping  or  duplicate  development  of  key  algorithms, 
concepts,  and  components. 

Inaccurate,  poorly  defined,  or  nonexistent  cost  and  schedule 
estimates  for  computer  resource  development. 

Multi-source  cost  and  schedule  estimates  using  a  variety  of 
estimating  techniques  and  models.  Avoid  basing  all  esti¬ 
mates  on  "lines  of  code"  estimates  derived  from  a  single 
source. 

Inadequate  developer  and  acquisition  manager  capability  or 
capacity  for  software  development. 

Inadequate,  immature,  or  poorly  integrated  software  develop¬ 
ment  tools  (e.g.,  compilers,  linkers,  loaders)  &  programming 
support  environment. 

Reviews  of  offerors  sites  to  assess  capability  and  capacity 
of  development  personnel,  management  structure  and 
procedures,  and  facilities. 

Lack  of  adequate  spare  computer  hardware  capacity  (e.g., 
processor  speed,  memory,  input/output,  and  secondary 
storage. 

Early  planning  for  spare  capacity  during  development  and 
support  phases  of  the  lifecycle;  periodic  reviews  of  capa¬ 
city  allocation,  and  projection  of  requirements  trends. 

Undefined  or  poorly  defined  software  support  concepts. 

Rigorous  adherence  to  the  separtion  of  mission  software 
and  system  softwaie  into  separate  CSCIs. 

Table  8-1.  Common  Risks  and  Possible  Corrective 


(e)  Insure  that  software  testing  is  adequately 
addressed  in  the  TEMP  and  monitor  com¬ 
pliance  as  the  program  matures; 

(f)  Identify  and  prioritize  the  required 
software  quality  factors  such  as  inter¬ 
operability,  portability,  flexibility,  useability, 
reusability,  maintainability,  integrity, 
reliability,  correctness,  testability,  and  ef¬ 
ficiency. 


(g)  Define  the  scope  of  the  IV&  V  effort  and 
develop  a  recommended  approach  using  con¬ 
tractor  or  government  personnel; 

(i)  Evaluate  the  use  of  standard  equipment, 
HOLs,  and  instruction  set  architectures; 

(j)  Evaluate  the  need  for  development  of 
software  tools  and  recommend  a  develop¬ 
ment  approach 
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8.5  SYSTEM  SECURITY 

The  Program  Manager  should  review  security 
directives  and  identify  mission  security  needs 
early  in  the  development  cycle.  Experience 
has  shown  that  it  is  much  cheaper  to  design 
security  into  the  system  from  the  outset  than 
to  add  it  on  to  a  mature  system.  It  is  more 
cost-effective  to  include  security  in  the  initial 
design.  A  security  requirements’  analysis 
should  be  conducted  even  if  funding  is  very 
limited  and  trusted  security  features  are  not 
incorporated  into  the  system  design. 

The  ability  to  evaluate  the  security  of  a  system 
improves  the  user’s  confidence  that  the 
security  mechanisms  are  sufficient  and 
functioning.  If  security  features  are  examined 
in  the  design  phase,  assessment  criteria  can  be 
established  in  advance,  and  security  features 
tested  and  evaluated  throughout  the  develop¬ 
ment  life  cycly  rather  than  at  system  deploy¬ 
ment.  This  also  makes  it  easier  to  satisfy  the 
system  certification  and  accreditation  re¬ 
quirements.  System  certifica  ion  ensures  that 
technical  computer  security  features  have 
been  tested  and  are  shown  to  be  adequate. 
System  accreditation  ensures  that  the  system 
safeguards  meet  DOD  operation  security  re¬ 
quirements.. 

Unfortunately,  MIL-STD-2167A  is  silent  on 
security  and  there  are  no  security  Data  Item 
Descriptions  referenced  in  that  standard. 
DOD-STD-5200.28,  DOD  Tnvited  Computer 
System  Evaluation  Criteria,  however,  provides 
evaluation  methods  and  criteria  for  system 
CLirity.  This  book,  generally  called  the 
orange  book"  and  published  by  the  National 
Computer  Security  Center  (NCSC).  is  sup¬ 
ported  by  several  related  security  guides.  This 
group  of  documents  is  informally  known  as 
the  "Rainbow  Series".  Security  provisions 
must  al.so  be  incorporated  into  planning  docu¬ 
ments,  such  as  the  PMP,TEMP,  Il^P,  and  the 


CRLCMP,  and  examined  during  the  various 
engineering  studies. 

It  is  beyond  the  scope  of  this  guide  to  provide 
complete  information  on  system  security; 
however,  the  PM  is  ultimately  responsible  for 
the  .security  of  system.  It  is  imperative  that  the 
PM  appoint  a  security  manager  early  in  the 
acquisition  cycle. 

8.6  CONTRACTUAL  CONSIDERATIONS 

A  recurring  activity  conducted  by  the  program 
office  is  that  of  selecting  one  or  more  contrac¬ 
tors  and  putting  them  on  contract.  Although 
the  intensity  of  the  activity  and  the  number  of 
contractors  to  be  evaluated  may  differ,  the 
process  of  selecting  a  contractor(s)  is  the  same 
regardle.ss  of  the  phase  of  development.  The 
three  major  activities  performed  are:  generat¬ 
ing  a  .source  selection  plan,  generating  a  re¬ 
quest  for  proposal  package,  and  conducting 
the  source  selection  process. 

8.6.1  Source  Selection  Plan  (SSP) 

An  outline  of  a  typical  SSP  is  given  in  Appen¬ 
dix  G.  The  SSP  is  a  key  document  for  initiat¬ 
ing  and  conducting  a  source  selection.  As 
shown  in  Appendix  G,  the  SSP  should  address 
mission  critical  computer  resources.  It  is 
prepared  by  the  program  office  and  must 
reflect  applicable  program  management 
direction  or  guidance.  For  the  Air  Force,  this 
guidance  is  reflected  in  the  Program  Manage¬ 
ment  Directive  (PMD).  The  SSP  is  a  plan  for 
organizing  and  conducting  the  evaluation  and 
analysis  of  proposals  and  a  roadmap  for  the 
.selection  of  a  source  or  sources. 

The  SSP  must  be  submitted  sufficiently  in 
advance  of  the  planned  acquisition  action  to 
facilitate  review  and  approval  by  the  Source 
Selection  Authority  (SSA)  and  early  estab¬ 
lishment  of  the  Source  Selection  Advisory 
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Council  (SSAC)  and  the  Source  Selection 
Evaluation  Board  (SSEB). 

8.6.2  Request  for  Proposal  Package  (RFP) 

The  RFP  package  must  be  sufficiently 
detailed  to  allow  responding  offerors  to  ade¬ 
quately  address  system  requirements  and  to 
provide  other  information  necessary  for 
evaluation  and  award.  The  RFP  package  typi¬ 
cally  consists  of  a  requirements  specifica- 
tion(s),  instructions  to  offerors,  proposal 
evaluation  criteria,  a  statement  of  work,  a 
work  breakdown  structure,  requirements  for 
deliverable  items,  and  special  contract  re¬ 
quirements.  Supporting  information  that  ex¬ 
pands  on  the  system  operations  and  support 
concepts,  including  the  CRLCMP,  may  be 
attached  to  the  RFP  package. 

Even  though  the  RFP  is  prepared  by  the  pro¬ 
gram  office,  it  is  good  practice  to  solicit  inputs 
from  the  using  and  supporting  organizations. 
They  usually  provide  valuable  insight  into  the 
operational  and  support  environment. 

8.6.2.1  Requirements  Specincation(s) 

The  requirements  specification(s)  included  in 
the  RFP  is  dependent  on  the  phase  of  the 
system  development  being  undertaken.  If 
one  is  contracting  for  the  CE  phase,  then  the 
specification  would  be  an  overall  system 
specification.  For  the  D/V  phase,  the 
specification  would  be  the  System/Segment 
Specification.  For  the  FSD  phase,  the 
specifications  would  be  the  refined  Sys¬ 
tem/Segment  Specification  and  could  include 
the  Software  Requirements  Specification  and 
the  Interface  Requirements  Specification. 
Appendix  H  lists  the  various  Data  Item 
Descriptions  (DIDs)  called  out  by  DOD- 
STD-2167A  which  govern  the  format  tmd 
content  of  the  required  specifications. 


8.6.2.2  In.structions  to  Oflerors 

In  addition  to  specifying  proposal  form  and 
content,  the  instructions  to  offerors  must  re¬ 
quire  submission  of  such  documents  as  a 
Software  Development  Plan,  a  Configuration 
Management  Plan,  and  a  Software  Quality 
Program  Plan  as  pat  t  of  the  proposal.  These 
plans  should  include  the  offerors’  software 
development  organization,  their  develop¬ 
ment  methodology,  their  management 
philosophy,  and  their  procedures  for  control¬ 
ling  and  assessing  development  progress.  Ap¬ 
pendix  H  lists  the  various  DIDs  which 
de.scribe  the  format  and  content  of  these 
documents. 

The  instructions  to  offerors  are  the 
mechanism  for  ensuring  that  offerors  address 
critical  software  issues  such  as: 

(a)  The  methodology  used  to  perform 
software  sizing  and  cost  estimating  and  the 
approach  to  be  followed  during  software 
development; 

(b)  The  rationale  used  for  the  computer 
resources  timing  and  sizing  estimates; 

(c)  A  description  of  any  teaming  and  sub¬ 
contractor  arrangements; 

(d)  The  skill  levels  required  for  computer 
resources  development  and  their  availability 
within  the  corporate  structure; 

(e) Thc  method  to  be  used  for  risk  control; 

(f)  Any  planned  use  of  firmware; 

(g)  Reusing  or  modifying  existing  software; 

(h)  A  clctir  definition  of  all  assumptions  used 
during  proposal  pre[)aration; 
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(i)  Plans  for  the  development  of  prototype 
software; 

(j)  Plans  and  procedures  for  generating  and 
using  software  metrics. 

(k)  The  computer  language  that  is  to  be  used 

(l)  The  level  of  system  security  required. 

8.6.23  Proposal  Evaluation  Criteria 

The  evaluation  criteria  must  be  based  on  the 
requirements  within  the  RFP.  This  includes 
computer  resource  development  and 
management  activities  and  the  offerors’ 
software  management  plans  described  in  the 
Software  Development  Plan  and  other  ap¬ 
plicable  documents.  The  criteria  in  the  RFP 
should  be  listed  in  relative  order  of  impor¬ 
tance.  The  evaluation  criteria  must  include 
the  availability  of  software,  documentation, 
and  the  rights  necessary  to  meet  life  cycle 
needs  and  the  compatibility  of  the  proposed 
design  with  the  support  concept  defined  in  the 
CRLCMP.  This  will  ensure  that  the  design  is 
modifiable  and  that  proposed  support  resour¬ 
ces  and  methods  are  adequate.  When  the 
processing  of  sensitive  or  classified  informa¬ 
tion  is  involved,  the  program  office  must  en¬ 
sure  that  computer  security  is  also  included  in 
the  evaluation  criteria. 

8.6.2.4  Statement  of  Work  (SOW) 

The  SOW  will  identify  the  applicable  program 
management,  development,  test,  training,  in¬ 
stallation  and  support  tasks  to  be  performed. 
More  specifically,  the  SOW  will: 

(a)  Identify  clear  and  concise  statements  of 
specific  task; 

(b)  Address  the  planned  use  of  an  Inde¬ 
pendent  Verification  and  Validation  contrac¬ 


tor  and  the  type  and  amount  of  support  ex¬ 
pected  from  the  development  contractor; 

(c)  Tailor  all  contra  .ly  required  standards 

and  specifications  to  the  program  needs; 

(d)  Address  the  planned  use  of  government 
provided  operational  and  environmental 
simulators,  support  equipment,  or  other 
software  programs  (e.g.,  compilers); 

(e)  Require  comprehensive  layout  of  pro¬ 
gram  schedules  to  include  reviews,  technical 
interchange  meetings,  audits,  and  testing; 

(f)  Address  the  requirements  for  prototype 
software  development; 

(g)  Address  the  requirements  for  generating 
and  using  software  metrics  data. 

(h)  Address  the  appropriate  trusted  system 
accreditation  and  certification  requirements. 

8.6.2.5  Work  Breakdown  Structure 

A  preliminary  work  breakdown  structure 
(WBS)  may  be  included  in  the  RF'P  package. 
The  contractor  will  be  expected  to  develop 
their  own  WBS  containing  additional  levels  of 
detail. 

8.6.2.6  Deliverable  Items 

Deliverable  computer  hardware  and 
software,  including  support  and  test  software, 
will  be  specified  as  contract  line  items 
(CLINs)  in  the  schedule  of  the  contract.  The 
CLINs  should  specifically  call  out  deliveries 
of  such  items  as  operational  flight  programs 
(OFPs),  test  program  sets  (TPSs),  simulation 
software,  and  incremental  deliveries  of 
various  versions  of  all  of  these.  Documenta¬ 
tion  requirements  will  be  identified  in  the 
Contract  Data  Requirements  List  (CDRL), 
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and  software  media  delivery  requirements 
will  be  specified  in  the  Software  Require¬ 
ments  Specification  which  will  be  listed  in  the 
CDRL.  For  software,  deliverable  items  will 
include  complete  source  code  in  a  form 
suitable  for  compilation  or  assembly  and  the 
complete  object  code  in  a  form  suitable  for 
loading  and  executing  in  either  operational  or 
support  computers.  The  CDRL  may  include 
the  documentation  needed  for  developing, 
testing,  operating,  and  supporting  the  system 
and  for  training  personnel.  Appendix  H  lists 
the  various  DIDs  for  this  additional  documen¬ 
tation.  If  all  the  documentation  needs  cannot 
be  identified  before  contract  award,  the 
CDRL  may  include  a  report  that  will  identify 
data  items  needed  to  satisfy  the  system  sup¬ 
port  and  operational  concepts.  A  Data  Ac¬ 
cession  List  should  be  used  to  identify  the 
contractor’s  informal  documentation  to  be 
made  available  for  government  review. 

8.6.2.7  Special  Contract  Requirements 

Special  requirements  are  tailored  contractual 
clauses  incorporated  into  the  contract  to  in¬ 
sure  the  government’s  right  to  computer 
software  and  to  provide  adequate  protection 
whenever  commercial  off-the-shelf  software 
is  used. 

Restricted  Rights  Software  -  The  RFP  should 
require  the  offeror  to  identify  and  cost  all 
restricted  rights  computer  software  or  equip¬ 
ment,  associated  documentation,  and  support 
items  required  to  be  delivered,  or  subject  to 
order,  under  the  contract. 

Commercial  Off-the-shelf  Software  (COTS)  - 

Procedures  may  be  developed  and  incor¬ 
porated  into  the  contract  to  ensure  liiai  the 
contractor  reviews  all  subcontractor  or  ven¬ 
dor  products  and  that  all  commercial 
hardware  and  software  in  the  system  are  sup¬ 
ported  to  the  correct  configuration  level.  The 


contractor  should  be  made  responsible  for 
maintaining  engineering  compatibility  be¬ 
tween  all  system  hardware  and  software,  in¬ 
cluding  the  incorporation  of  newly  released 
versions  of  software.  Operating  system 
software  falls  into  this  category. 

8.6.3  Source  Selection  Process 

The  principal  objective  of  the  source  selection 
process  is  to  select  the  source  (offeror)  whose 
proposal  has  the  highest  degree  of  credibility 
and  whose  performance  can  be  expected  to 
best  meet  the  government’s  requirements  at 
an  affordable  cost.  The  process  should  pro¬ 
vide  an  impartial,  equitable,  and  comprehen¬ 
sive  evaluation  of  the  competitors’  proposals 
and  related  capabilities.  The  process  should 
be  accomplished  with  minimum  complexity 
and  mtiximum  efficiency  and  effectiveness.  It 
should  be  structured  to  properly  balance  tech¬ 
nical,  financial,  and  economic  considerations 
consistent  with  the  phase  of  the  acquisition, 
program  requirements,  and  business  and  legal 
constraints,  A  typical  source  selection  process 
is  depicted  in  Figure  8-1. 

8.6.3.1  Draft  RFP 

If  it  is  at  all  possible,  the  process  should  begin 
with  a  draft  RFP  which  consists  of  preliminary 
versions  of  a  statement  of  work,  a  specifica¬ 
tion,  schedules,  a  contract  data  requirements 
list  (CDRL),  and  evaluation  criteria.  Unfor¬ 
tunately,  time  and  resources  do  not  always 
permit  a  program  office  to  generate  and  cir¬ 
culate  a  draft  RFP. 

By  circulating  a  draft  RFP  among  the  various 
internal  government  organizations  (i.e.,  con¬ 
tracts,  legal)  the  RFP  can  be  evaluated  for 
consistency  and  content.  These  organizations 
can  provide  valuable  comments  which  help  to 
ensure  that  the  final  RFP  doesn’t  become 
bogged  down  because  of  major  shortcomings. 


8-9 


Chapter  8  Planning  for  Computer  Software 


Fig.  8-1  Source  Selection  Process 


An  industry  review  can  also  be  very  helpful. 
Industry  may  provide  constructive  informa¬ 
tion  on  the  potential  technical,  schedule,  and 
cost  risks  associated  with  the  intended 
procurement.  For  example,  they  may  provide 
alternative  approaches  to  high  risk  areas  and 
indicate  areas  which  are  ambiguous,  con¬ 
tradictory,  or  likely  to  be  major  cost  drivers. 


ing  the  source  selection  organization  and  the 
proposal  evaluation  criteria.  Although  the 
general  evaluation  criteria  have  already  been 
released  with  the  RFP,  specific  sub-criteria 
and  factors  may  still  require  fine  tuning 

8.6.3.2  Populating  the  Source  Selection  Or¬ 
ganization 


All  industry  comments,  however,  should  be 
carefully  examined  to  separate  fact  from 
marketing  information.  Some  companies 
may  not  be  able  to  resist  the  temptation  to 
suggest  changes  or  additions  to  the  RFP  which 
could  enhance  their  competitive  posture.  In 
spite  of  this  danger,  industry  comments  can  be 
tremendously  useful. 

A  final  RFP  can  be  prepared  and  released 
once  comments  have  been  received  from  in¬ 
dustry  and  other  internal  organizations. 
While  the  bidders  are  preparing  their 
proposals,  the  program  office  will  be  finaliz¬ 


One  of  the  constant  problems  faced  by  pro¬ 
gram  managers  is  finding  qualified  software 
individuals  willing  and  able  to  be  away  from 
their  jobs  for  an  extended  period  of  time. 
Source  selections  can  take  anywhere  from 
three  to  six  months  to  complete.  Since  quality 
software  individuals  are  already  in  short  supp¬ 
ly,  a  program  manager  may  have  to  resort  to 
innovative  means  for  acquiring  them.  If  one 
or  two  experienced  software  people  are  al¬ 
ready  on  the  staff,  then  the  problcui  becomes 
more  manageable.  The  task  of  finding  addi¬ 
tional  knowledgeable  but  less  experienced 
software  people  is  a  bit  easier. 
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DETAILED 
PROPOSAL  EVAL 


Fig.  8-2  Evaluation  Process 


ITie  problem  is  more  difficult  if  a  program 
manager  does  not  have  in-house  expertise. 
The  first  place  to  look  for  experienced 
software  personnel  is  in  another  program  of¬ 
fice  that  may  be  willing  to  release  in- 
dividual(s)  for  the  duration  of  the  source 
selection.  Another  source  could  be  the 
various  laboratories  within  the  command  or 
service.  A  sister  service  may  be  able  to  pro¬ 
vide  temporary  software  expertise.  There  are 
also  federally  established,  not-for-profit  or¬ 
ganizations  such  ixi  .  Aerospace  Corp.,  the 
Mitre  Corp.,  and  the  Software  Engineering 
Institute  whose  employees  routinely  par¬ 
ticipate  in  government  source  selections. 
Other  corporations  exist  which  may  be  able  to 
participate  in  a  source  selection.  These  cor¬ 
porations  are  primarily  analyses  oriented  and 
normally  do  not  produce  or  develop  hardware 
for  the  commercial  market.  The  Rand  Corp 
and  the  Charles  Draper  Laboratory  fall  into 
this  category.  The  lead  software  individual, 
however,  must  be  a  government  employee. 


8.6  J.3  Evaluation  Process 

The  evaluation  process  itself  is  well  defined 
and  regulated  by  the  DOD  and  service 
peculiar  regulations  and  procedures.  Figure 
8-2  depicts  the  typical  evaluation  process 
which  begins  with  the  receipt  of  cost,  technical 
and  management  proposal  volumes,  as  a  min¬ 
imum.  The  Source  Selection  Evaluation 
Board  (SSEB)  is  comprised  of  the  functional 
area  experts  who  actually  perform  the 
detailed  evaluation.  Since  this  is  a  time-con¬ 
suming  but  critical  step,  it  is  important  that 
the  lead  individual  expedite  the  process. 
Endless  discussions  over  trivial  or  irrelevant 
points  cannot  be  tolerated.  Even  when 
serious  topics  are  the  source  of  major  dis¬ 
agreements  among  the  evaluators,  the  lead 
software  individual  must  force  the  people  in¬ 
volved  to  come  to  a  reasonable  and  timely 
consensus.  Minority  opinions  and  views 
should  be  aired  and,  at  the  discretion  of  the 
lead  software  person,  documented  and 
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presented  to  the  Source  Selection  Advisory 
Council  (SSAC)  for  resolution.  By  the  same 
token,  an  individual’s  infatuation  with  a  par¬ 
ticular  technology  should  not  be  allowed  to 
unduly  bias  the  proposal  ratings.  For  ex¬ 
ample,  an  individual’s  infatuation  with  object- 
oriented  design,  should  not  interfere  with  his 
appraisal  of  a  more  conventional  approach 
(e.g.,  functional  decomposition). 

Once  the  SSEB  completes  its  evaluation,  the 
results  are  presented  to  the  SSAC.  The 
SSAC,  which  is  composed  of  senior  govern¬ 
ment  personnel,  then  evaluates  the  analysis 
and  findings  of  the  SSEB  and  presents  the 
results  to  the  Source  Selection  Authority 
(SSA).  The  SSA  is  the  official  designated  to 
direct  the  source  S(. lection  and  to  make  the 
final  source  selection  decision. 

8.6.3.4  Evaluating  Offeror’s  Proposal 

Since  every  program  has  unique  require¬ 
ments,  it  is  beyond  the  scope  of  this  guidebook 
to  provide  specific  information  on  what  is 
important  in  a  software  source  selection.  In 
general,  however,  the  software  evaluation  as¬ 
sesses  the  technical  adequacy  of  the  proposed 
computer  system  architecture  to  satisfy  the 
weapon  system  requirements.  Items  that  are 
evaluated  include: 

(a)  The  throughput  and  memory  capability  of 
the  proposed  computers; 

(b)  Future  vendor  support  for  commercially 
supplied  items  such  as  tape  drives,  disk  drives, 
controllers,  etc.; 

(c)  Computer  resources  interfaces  to  the  rest 
of  the  system  architecture  and  human 
operators; 

(d)  Adequacy  of  the  operating  system  or 
software  executive; 


(e)  Availability,  currency,  and  usage  of 
software  development  tools  and  methods; 

(f)  Organic  supportability  of  computer 
hardware  and  software; 

(g)  The  offeror’s  Software  Development  Plan 
and  software  development  standards  and  pro¬ 
cedures; 

(h)  The  offeror’s  software  development 
capability  and  capacity. 

It  is  important  to  emphasize  the  need  to  per¬ 
form  an  integrated,  comprehensive  evalua¬ 
tion  of  the  offerors’  total  proposal.  This 
usually  means  that  technical  evaluators  must 
also  review  the  management  and  cost 
proposals.  The  cost  evaluators  normally  con¬ 
centrate  on  accounting  and  costing  consisten¬ 
cy  and  completeness.  They  are  not  qualified 
to  pass  judgment  on  whether  a  proposed  num¬ 
ber  of  man-hours  are  sufficient  for  a  par¬ 
ticular  analysis  or  effort.  Only  the  technical 
evaluators  can  make  this  assessment. 
Likewise  the  technical  evaluators  can  also  as¬ 
sess  whether  a  particular  management  or¬ 
ganization  or  procedure  is  consistent  with  the 
technical  effort  proposed.  One  is  not  advocat¬ 
ing  that  all  technical  evaluators  review  cost 
and  management  data.  What  is  being 
proposed  is  that  one  or  two  key  individuals 
from  the  technical  panel  make  a  top-level 
review  of  cost  and  management  data  for 
realism  and  consistency. 

8.6.3.5  Software  Development  Capability 
Capacity  Review 

Figure  8-1  shows  a  software  development 
capability  and  capacity  review  (SDCCR)  oc¬ 
curring  as  part  of  the  source  selection  process. 
The  SDCCR  has  been  successfully  used  at  the 
Aeronautical  Systems  Division  of  the  Air 
Force  Systems  Command  at  Wright-Patter- 
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son  AFB.  It  is  described  in  ASD  Pamphlet 
800-5,  Software  Development  Capability  and 
Capacity  Review  [1]. 

Purpose  -  The  SDCCR  is  intended  to  review 
and  assess  an  offeror’s  specific  capability  and 
capacity  to  develop  the  software  required  on 
a  particular  weapon  system  program  as 
defined  in  the  RFP.  This  review  process  is 
designed  to  be  incorporated  as  an  integral 
part  of  the  FSD  source  selection  process.  The 
review  process  accomplishes  three  related 
objectives.  First,  the  acquisition  manage¬ 
ment  team  gains  an  understanding  of  the 
offeror’s  software  development  methods  and 
tools.  Second,  the  capability  and  capacity  of 
the  offeror  to  develop  the  required  software 
in  a  disciplined  software  development  process 
is  determined.  Third,  the  review  process 
elicits  a  contractual  commitment  by  the  of¬ 
feror  to  implement  the  methods,  tools,  prac¬ 
tices  and  procedures  which  form  the 
discipline  and  structure  for  this  software 
development  process  [1]. 

Process  Summary  -  The  SDCCR  process  is 
accomplished  during  the  FSD  RFP  prepara¬ 
tion  and  source  selection  phase.  The  RFP 
includes  the  requirement  that  the  offeror 
team  provides  specific  information  describing 
their  software  development  methods,  and  in¬ 
clude  examples  of  how  the  methods  have  been 
applied  on  past  or  on-going  programs.  The 
SDCCR  source  selection  team  reviews  this 
information  and  then  conducts  an  in-plant 
review  with  the  offeror’s  team.  This  review  is 
based  on  a  specific  set  of  SDCCR  questions 
which  are  provided  with  the  RFP  to  the  of¬ 
ferors  and  are  found  in  Attachment  4  of  ASD 
Pamphlet  800-5.  Following  this  one  to  two 
day  in-plant  review,  the  offeror’s  capability 
and  capacity  to  develop  the  required  software 
is  assessed  using  the  predefined  RFP  stand¬ 
ards.  This  evaluation  becomes  an  integral  part 
of  the  program  source  selection  and  forms 


part  of  the  basis  for  the  award.  It  is  highly 
recommended  that  offerors  conduct  this 
review  with  their  subcontractors  prior  to  the 
government’s  in-plant  review  [1], 

Review  Areas  and  Factors  -  The  SDCCR  is 
usually  organized  into  five  major  areas: 
management  approach,  management  tools, 
development  process,  personnel  resources, 
and  Ada  technology.  These  areas  are  in  turn 
organized  into  factors  as  shown  in  Table  8-2. 
Other  factors  may  be  added  to  this  review  as 
a  function  of  unique  program  requirements. 

Team  Composition  -  The  SDCCR  is  per¬ 
formed  by  the  source  selection  team.  This 
approach  is  fundamental  to  achieving  the 
multiple  objectives  of  the  SDCCR.  The  usual 
team  composition  is  as  follows: 

(a)  Team  Chief  -  Computer  resources  systems 
engineer  or  senior  software  engineer  from  the 
engineering  staff; 

(b)  Program  Manager/Project  Manager; 

(c)  Software  Manager; 

(d)  Software  Engineer; 

(e)  Contracting  Officer; 

For  smaller  programs,  it  is  possible  for  one 
individual  to  perform  the  program/project 
and  software  management  role,  and  one  in¬ 
dividual  to  perform  the  chief/lead  and 
software  engineering  role.  In  addition,  it  is 
desirable  to  include  the  following  participa¬ 
tion  on  the  team: 

(a)  Product  and  Quality  Assurance; 

(b)  Configuration  Data  Management; 

(c)  Logistics; 
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(d)  Cost  Analyst; 

(e)  Defense  Contract  Management  Com¬ 
mand  (DCMC)  personnel. 

It  is  recognized  that  too  large  a  team  is 
counterproductive.  The  key  is  to  perform  the 
review  with  a  small,  knowledgeable  group  of 
software  experienced  acquisition  personnel. 

8.6.3.6  Software  Capability  Evaluation 

Another  method  of  assessing  the  contractor’s 
software  engineering  capability  is  the 
Software  Capability  Evaluation  (SCE).  The 
method  has  been  used  in  competitive  bids  for 
programs  at  the  Naval  Air  Development  Cen¬ 
ter,  the  Air  Force  Electronic  Systems 

MANAGEMENT  APPROACH 

-  Management  Organization 

•  Software  Management  System 

-  Software  Configuration  Management 

-  Software  System  Organizations  and  Structure 

•  Softwae  Subcontracting 

-  Software  Pianning 

•  Software  Quality/Product  Assurance 

•  Contract  Control  Methods 

MANAGEMENT  TOOLS 

-  Internal  Management  Standards  and  Tools 

-  Software  Size,  Manpower,  Schedule,  and  Cost 

Estimating 

-  Contract  Work  Breakdown  Structure  (CWBS) 

-  Software  Work  Definition 

-  Schedule  Definition  and  Statusing 

-  Software  Cost  Performance  Reporting  System 

DEVELOPMENT  PROCESS 

-  Internal  Development  Standards  and  Procedures 

-  Software  Engineering 

•  Software  Development  Tools  and  Facilities 

•  Software  Test  and  Verification 

-  Software  Documentation  Approach 

-  Internal  Independent  Verification  and  Validation 

PERSONNEL  RESOURCES 

-  Estimating  Software  Personnel  Requirements 

-  Manpower  Needs  and  Qualifications 

•  Managing  Software  Personnel  Resources 

•  Company  Workload  Profile 

Ada  TECHNOLOGY 

-  Management  Process 

-  Development  Process  and  Environment  (Tool  Set) 

-  Design  Process  and  Methodology 

-  Personnel  Skills  and  Qualifications 

-  Capability  Demonstration  and  Risk  Management 


Division,  and  the  Army  Communications  and 
Electronics  Command  [2]. 

Purpose  -  The  SCE  is  the  Software  Engineer¬ 
ing  Institute’s  (SEI’s)  method  for  assessing  a 
contractor’s  software  engineering  capability 
[2].  It  is  used  during  the  source  selection  and 
contract  monitoring  phases  of  software-inten¬ 
sive  or  software-critical  system  acquisitions. 

The  SCE  augments  the  acquisition  process  by 
determining  a  contractor’s  strengths  and 
weaknesses  with  respect  to  a  maturity  model. 
In  addition,  it  establishes  "software  capability" 
as  a  criterion  for  source  selection  by  providing 
an  orderly  way  of  comparing  offerors’ 
software  capability  against  a  standard  set  of 
criteria.  This  is  an  important  advantage.  This 
method  should  be  used  to  augment  current 
source  selection  and  contract  monitoring 
software  risk  assessment  steps. 

The  SEI,  with  help  from  the  MITRE  Corpora¬ 
tion,  developed  this  method  for  the  U.  S.  Air 
Force.  It  was  motivated  by  the  increasing 
importance  of  software  in  DOD  acquisitions 
and  the  need  for :  he  services  to  evaluate  more 
effectively  software  contractors’  abilities  to 
perform  software  engineering. 

The  SEI  is  striving  toward  two  major  goals  in 
this  effort: 

(a)  providing  a  standardized  software  process 
evaluation  method  which  is  documented, 
available  for  review  and  comment,  and  peri¬ 
odically  modified  as  experience  is  gained  with 
its  use; 

(b)  making  that  evaluation  a  public  process 
which  is  defined  in  advance  and  for  which 
contractors  can  prepare. 

By  enabling  a  consistent  measurement  of 
contractors’  software  development  capability. 


Table  8-2  SDCCR  Factors 
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the  SCE  method  should  improve  contractor 
and  acquisition  management. 

Process  Summary  -  The  Program  Manager 
and  the  Procuring  Contracting  Officer  in¬ 
clude  SCE  wording  in  the  Source  Selection 
Plan  and  the  RFP.  They  also  ensure  that 
proper  training  in  SCE  is  provided  to  acquisi¬ 
tion  personnel  and  the  Capability  Evaluation 
Team  (CET).  In  their  proposals  the  offerors 
will  respond  to  an  SCE  questionnaire  and 
provide  a  list  of  candidate  projects,  in  the 
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Fig.  8-3  SCE  in  the  RFP 

form  of  "project  profiles",  for  evaluation 
during  the  CET’s  In-Plant  Review  (Figure 
8-3)  [3]. 

After  the  CET  reviews  the  responses  to  the 
questionnaire  and  the  listing  of  candidate 
projects,  four  projects  are  selected  for  the 
In-Plant  Review.  An  agenda  and  any  requests 
for  further  documentation  are  sent  to  the  of¬ 
feror  before  the  CET  visit  (Figure  8-4).  After 


/  Capability 
I  Evaluation 


.vaiuauuii  > - 1 - - - 1 - 

Team  /[  Agen"^ 


Offerors 


Project  list  for 
In-Plant  Review 


Initial  SCE 
Response 


Documentation 

Requests 


In-Plant 

Review 

Meeting 


Requested 

Documen- 


izi'z:' 

SCE 

Findings 


Fig.  8-4  SCE  in  the  In-Plant  Review 

the  In-Plant  Review  the  CET  will  submit  a 
report  of  their  findings  for  the  SSAC.  The 
report  addresses  key  software  areas  such  as 
project  management,  software  quality  as¬ 
surance,  and  configuration  management  (Fig¬ 
ure  8-5). 
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CHAPTER  9 


MANAGEMENT  PRINCIPLES 


9.1  INTRODUCTION 


Managing  software  development  is  one  of  the 
biggest  challenges  facing  today’s  government 
program  manager  (PM),  The  classic 
problems  contributing  to  this  challenge  are 
illustrated  in  Figure  9-1,  Some  of  these  pro¬ 


blems  are  beyond  the  control  of  the  PM.  For 
example,  there  is  very  little  a  PM  can  do  about 
changing  or  growing  requirements  because  of 
an  evolving  threat,  or  fuzzy  requirements  be¬ 
cause  of  lack  of  details  on  the  threat.  Long 
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lead  times  often  force  systems  development 
to  be  initiated  before  the  threat  has  been 
completely  evaluated.  Shooting  at  a  moving 
target  of  changing  or  evolving  requirements  is 
a  reality  of  all  new  systems  acquisitions. 

There  are,  however,  many  problems  which 
can  be  minimized  or  eliminated  by  the  PM. 
For  example,  there  are  certain  actions  that 
can  be  taken  by  a  PM  to  eliminate  poor  en¬ 
gineering  discipline,  poor  documentation,  or 
poor  planning.  This  chapter  will  address 
methods  and  principles  which  can  help  a  PM 
deal  with  the  complexity  and  difficulty  of  large 
scale  software  system  development.  In  deal¬ 
ing  with  these  problems,  the  common 
denominator  is  a  disciplined  system  engineer¬ 
ing  approach  incorporating  the  principles  of 
software  engineering. 

9.2  SOFTWARE  ENGINEERING 

As  mentioned  in  Chapter  2,  the  term  software 
engineering  is  relatively  new.  Today  there  is 
some  debate  as  to  what  constitutes  software 
engineering,  but  most  people  agree  that  ap¬ 
plying  the  term  software  engineering  to  the 
software  development  process  implies; 

(a)  The  application  of  proven  methods  at  each 
step  of  the  process,  including  accepted  prac¬ 
tices,  standards  and  procedures; 

(b)  The  development  and  use  of  software 
tools  and  aids; 

(c)  The  generation  of  specific  documents 
during  the  various  stages  of  the  development 
process; 

(d)  A  traceable  path  from  the  system  require¬ 
ments  down  to  the  final  deliverable  product. 

For  these  proven  methods  to  be  effective  in 
controlling  software  development,  they  must 


be  accepted  and  practiced  by  the  software 
developer.  Mere  proposal  promises  and  con¬ 
tractual  language  are  not  sufficient.  To 
achieve  quality  software  which  performs  to 
specifications,  is  reliable,  and  is  reasonably 
priced,  these  methods  must  be  ingrained  in 
the  contractor’s  software  organization.  Table 
9-1  lists  some  of  the  proven  methods  for 
software  development. 

All  engineering  disciplines  develop  tools  and 
aids  to  help  the  practitioner  apply  engineering 
theory.  Because  software  engineering  is  a 
relatively  young  discipline,  it  doesn’t  have  the 
wealth  of  tools  so  readily  available  to  its  older 
sister  disciplines.  Much  progress  is  being 
made,  however,  and  currently  available  tools 
and  aids  include: 

(a)  Stiuctured  design  and  programming; 

(b)  Inspections  and  walkthroughs; 

(c)  Computer  aided  software  engineering; 

(d)  Program  design  languages; 

•  Visible  Training  and  Education  Program 

•  Inslitutionalieed  Practices  and  Procedures 

•  Defined  Roles  With  Systems,  Hardware,  and 
Test  Organizations 

•  Adherence  to  a  Software  Development  Plan 

•  Institutionalized  Internal  Inspection  Procedures 

•  Structured  Design  and  Programming 

•  Product  Orienterf  WBS  for  Financial  Control 

•  Internal  Independent  Verification  and  Validation  (IV&V) 

•  Strong  Configuration  Management 

•  Software  Development  Library 
-  Problem/T rouble  Reporting  System 

•  Visible  Status  Information 

Table  9-1  Proven  Methods  for  Software 
Development 
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(e)  The  Ada  programming  language; 

(f)  Object  Oriented  Design  and  Program¬ 
ming. 

Documenting  each  step  of  the  software 
development  process  is  absolutely  essential. 
Not  only  does  one  need  an  effective  way  to 
communicate  the  rate  of  progress,  one  also 
needs  a  well-documented  final  product. 
Since  documentation  is  the  only  tangible 
evidence  of  the  resulting  product,  documen¬ 
tation  must  be  generated  as  one  progresses 
through  the  various  stages  of  software 
development.  One  cannot  do  a  very  effective 
job  of  generating  software  documentation 
after  the  fact. 

One  must  have  a  clearly  defined  path  from  the 
top  level  system  requirements  down  to  the 
various  modules  or  units  of  code.  This  en¬ 
sures  that  the  delivered  product  satisfies  the 
system  requirements.  It  can  be  accomplished 
through  the  use  of  a  requirements’  matrix 
which  shows  how  each  system  level  require¬ 
ment  is  satisfied  by  a  particular  module  or 
segment  of  code.  This  process  can  be 
automated. 

93  GVIDELINES  AND  RULES 

Over  the  past  30  years,  certain  guidelines  and 
rules  have  evolved  to  help  a  PM  successfully 
complete  a  software  development  program. 
One  must  keep  in  mind  that  the  guidelines 
and  rules  given  below  are  not  just  recipes  or 
checklists  to  be  blindly  followed  in  order  to 
ensure  success.  PMs  must  understand  how  to 
customize  these  guidelines  and  rules  to  the 
requirements  of  the  program.  They  may 
choose  to  ignore  any  one  of  them  as  long  as 
they  are  aware  of  the  risks  and  have  planned 
for  dealing  with  them.  The  following 
guidelines  provide  the  PM  with  software 
development  planning  and  organizational 


principles  which  place  software  development 
in  the  perspective  of  the  overall  picture  of 
system  development; 

(a)  When  planning  and  directing  a  program, 
PMs  must  make  decisions  based  on  a  "system" 
perspective.  They  must  review  their  alterna¬ 
tives  and  not  allow  either  hardware  or 
software  to  exclusively  drive  their  decision. 
All  decisions  must  consider  long  term  system 
effects. 

(b)  PMs  must  provide  a  forum  for  integrating 
the  system  development.  Several  techniques 
are  available  and  should  be  part  of  both  the 
government’s  and  the  contractor’s  review 
process.  These  techniques  include  stringent 
interface  controls,  reviews,  and  audits.  All 
interfaces  between  software  modules  and  be¬ 
tween  software  and  hardware  must  be  clearly 
defined  and  strenuously  controlled.  Techni¬ 
ques  such  as  reviews  and  audits  will  be  dis¬ 
cussed  in  the  next  section. 

(c)  Large  investments  up  ^ront  can  have  sig¬ 
nificant  leverage  on  reducing  later  system 
operation  and  support  costs.  Investing  resour¬ 
ces  during  the  system  and  software  require¬ 
ments  analysis  phases  can  result  in  a  better 
understanding  of  the  user’s  requirements  and 
a  more  stable  baseline  for  design.  Early  ex¬ 
penditure  of  resources  also  provides  the 
greatest  amount  of  leverage  in  preventing  er¬ 
rors.  The  bulk  of  the  errors  (30%  to  70%)  can 
be  detected  during  the  time  when  error  cor¬ 
rection  is  the  cheapest  [2]. 

(d)  Software  resource  planning  must  remain 
stable  once  the  program  starts.  Software 
development  has  an  inherent  resource  and 
schedule  profile  which  means  that  the  overall 
schedule  cannot  be  stretched  out  without  ad¬ 
versely  affecting  software  development. 
More  important,  squeezing  the  schedule  can 
be  a  prescription  for  disaster.  It  is  not  always 
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possible  to  add  manpower  to  solve  a  software 
schedule  slip  [3], 

(e)  The  software  architecture  should  be  the 
major  driver  in  determining  hardware  par¬ 
titioning.  In  the  past,  PMs  paid  more  atten¬ 
tion  to  hardware  because  they  understood  it 
best.  Today  the  PM  must  consider  the  real¬ 
time  requirements  of  the  system  and  whether 
parallel  processing  and/or  distributed 
processing  will  meet  the  overall  require¬ 
ments.  The  software  architecture  will  often 
drive  the  hardware  architecture. 

(f)  Identification  of  the  software  architecture 
should  be  performed  simultaneously  with  the 
requirements  definition  and  systems  analysis 
tasks.  To  accomplish  adequate  hardware  and 
software  tradeoffs,  the  software  must  be 
viewed  not  only  as  the  system  integrator  but 
as  a  system  in  its  own  right.  Software  develop¬ 
ment  is  usually  in  the  critical  path  of  the  over¬ 
all  system  development. 

(g)  The  PM  should  stimulate  innovation  and 
not  be  stifled  by  rules  and  regulation. 
'Thought  first,  regulations  second"  should  be 
the  theme  pursued  by  all  system  managers. 
New  approaches  should  not  be  surreptitiously 
dismissed.  DOD-STD-2167A,  "Defense  Sys¬ 
tem  Software  Development,"  is  only  a  start  and 
better  approaches  will  evolve.  This  is  not  to 
say  that  f3oD-STD-2167A  stifles  innovation. 
Although  the  standard  embraces  the  "water¬ 
fall"  approach  to  the  software  development 
process,  it  does  allow  for  other  advances  such 
as  rapid  prototyping,  evolutionary  develop¬ 
ment,  and  other  process  models.  In  practice, 
the  waterfall  model  is  best  suited  for  "prece- 
dented"  systems  (i.e.,  systems  that  have  been 
developed  at  least  once).  For  "unprece¬ 
dented"  systems,  or  systems  that  are  totally 
new,  other  models  such  as  Boehm’s  "spiral" 
model  may  be  more  appropriate. 


(h)  PMs  must  plan  for  growth  and  evolution. 
Experience  has  shown  that  the  user  cannot 
initially  handle  all  of  the  new  functions  at 
once,  even  if  the  developers  can  deliver  it  [4]. 
Fox  cites  the  example  of  the  Apollo  command 
and  control  system  developed  by  IBM.  When 
the  program  manager  was  asked  why  there 
were  over  fourteen  (14)  releases  to  the 
software,  he  replied  that  there  were  two  major 
reasons  for  this.  First,  the  hundreds  of 
operators  monitoring  consoles  and  interact¬ 
ing  with  the  computer  could  not  absorb 
operating  procedures  in  big  doses  -  they  had 
to  absorb  them  a  piece  at  a  time.  Capability  is 
best  provided  piecemeal.  Second,  they  could 
not  predict  very  far  in  advance  what  the  users 
would  want  or  what  would  be  needed  in  the 
control  system  [4].  The  software  architecture 
and  implementation,  therefore,  should  be 
specifically  pointed  toward  maximum 
modularity,  changeability,  and  growth  poten¬ 
tial.  The  PM’s  primary  concern  should  be  life 
cycle  cost.  Post  deployment  software  support 
needs  to  be  examined  early  during  develop¬ 
ment  to  provide  for  cost  effective  lifetime 
support. 

(i)  The  procurement  process  should  allow  for 
a  flexible,  robust,  and  expandable  software 
design.  Ways  must  be  found  to  reward  inno'-a- 
tive  contractors.  Some  suggestions  are  to 
consider  award  fees  or  cost  incentives  based 
on  post-delivery  operation  and  support.  The 
fees  or  incentives  could  be  based  on  the  ability 
to  perform  the  intended  functions,  ease  of 
support,  ease  of  modification,  utility  of 
documentation,  and  effectiveness  of  the 
human  interfaces.  The  intent  should  be  to 
force  the  developer  to  focus  on  long  term 
goals  and  a  supportable  system.  Also  impor¬ 
tant,  but  more  difficult  to  identify,  are  the 
front  end  decisions  and  methods  that  achieve 
these  goals.  What  the  developer  does  in  the 
beginning  has  the  most  influence  on  the  end 
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Is  a  mechanism  used  for  ensuring  traceability  between  the  software  top-level  and  detailed  designs? 

Are  internal  software  design  reviews  conducted? 

Is  a  mechanism  used  for  controlling  changes  to  the  software  design? 

Is  a  mechanism  used  for  ensuring  traceability  between  detailed  design  and  code? 

Arc  formal  records  maintained  of  unit  (module)  development  progress? 

Are  software  code  reviews  conducted? 

Is  a  mechanism  used  for  controlling  changes  to  the  code? 

(Who  can  make  changes  and  under  which  circumstances?) 

Is  a  mechanism  used  for  configuration  management  of  the  software  tools  used  in  the 
development  process? 

Is  a  mechanism  used  for  verifying  that  the  samples  examined  by  the  software  QA  are  truly 
representative  of  the  work  performed? 

Is  there  a  mechanism  for  assuring  the  regression  testing  is  routinely  performed? 

Is  there  a  mechanism  for  assuring  the  adequacy  of  regression  testing? 

Table  9-2  Process  Control 


product.  The  developer  must  begin  with  a 
good  process  already  in  place. 

9.4  PROCESS  CONTROL 

Process  control  is  the  key  to  achieving 
software  quality.  The  process  is  the  method 
used  by  the  contractor  for  developing 
software.  Achieving  control  of  the  process 
means  that  the  process  is  predictable  and 
measurable.  A  controlled  process  will  mini¬ 
mize  variability.  How  does  a  PM  assess  the 
contractor’s  process  control  system  and  pro¬ 
cedures?  Table  9-2  provides  a  series  of  ques¬ 
tions  developed  by  the  Software  Engineering 
Institute  [.5]  as  an  aid  in  making  this  assess¬ 
ment.  The  PM  must  know  that  the  process 
represents  the  contractor’s  commitment, 


philosophy,  methodology,  procedures,  and 
standards  for  doing  business.  Process  control 
and  process  management  are  principles  of  the 
Deming  philosophy  [6]  for  customer  satisfac¬ 
tion.  Application  of  Deming’s  philosophy  re¬ 
quires  the  commitment  of  top  management. 
That  is  why  it  is  so  important  to  select  the  right 
contractor;  these  values  are  not  learned  over¬ 
night. 

What  follows  are  some  program  management 
guidelines  which  focus  on  the  day-to-day 
management  of  software  development. 

All  Software  Tasks  Must  be  Discrete  -  This 
guideline  is  fundamental  in  determining  how 
well  the  contractor  can  plan  the  effort.  PMs 
must  not  allow  level  of  effort  or  percentage 
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complete  approaches  because  this  closes  the 
door  on  program  progress  visibility.  The  con¬ 
tractor  must  be  able  to  define  the  work  pack¬ 
ages  associated  with  the  work  breakdown 
structure  (WBS)  in  sufficient  detail  to  control 
and  manage  the  effort.  Each  task  should  have 
a  definite  start,  a  definite  end  date,  and  a 
specific  output.  These  tasks  are  normally  on 
the  order  of  30  to  90  man-days  in  duration. 
Planning  is  usually  accomplished  as  a  rolling 
wave,  the  immediate  six  months  or  more  are 
planned  in  detail  with  the  remaining  effort 
generally  only  visible  at  higher  levels  in  the 
WBS.  The  entire  effort  should  be  totally 
scoped  in  time  and  resources  at  the  very 
beginning  of  the  project.  If  the  program  un¬ 
certainties  are  too  great,  this  may  not  be  pos¬ 
sible. 

Quantitative  Requirements  are  Managed 
Through  Margins  -  Quantitative  require¬ 
ments  lend  themselves  to  measurable  control 
methods.  Computer  memory  and  through¬ 
put,  for  example,  are  often  tracked  during 
development.  In  the  beginning,  estimates  of 
software  size  and  timing  will  be  made  and 
compared  against  target  values  to  determine 
margins.  Later,  design  language  estimates 
can  be  made  and  these  estimates  will  continue 
through  the  design  process.  code  is  writ¬ 
ten,  actual  measurements  can  be  made.  Early 
planning  should  allow  for  contractor  and 
government  margins  with  the  use  of  a  dis¬ 
ciplined  and  documented  control  system. 
One  approach  is  to  baseline  an  estimate  of  the 
memory  and  throughput  utilization  on  a 
monthly  basis.  An  alert  or  trigger  threshold 
value  can  initiate  action  should  the  threshold 
values  be  exceeded. 

Identify  and  Track  Risk  Areas  -  The  contrac¬ 
tor  and  PM  should  be  working  as  a  team  to 
manage  program  risk.  An  important  in¬ 
gredient  of  any  program  is  to  assess  and 
reduce  the  risk  as  early  as  possible  and  before 


Milestone  II.  Risk  management  is  an  ongoing 
process.  The  first  step  is  to  identify  risk  areas, 
document  them  in  the  Software  Development 
PL.n,  and  devise  a  scherne  for  dealing  with 
each  risk  item.  These  items  are  then  tracked 
throughout  development.  A  convenient 
method  is  to  have  a  'Top  Ten"  list  of  risk  areas 
that  are  tracked  at  least  on  a  monthly  basis 
along  with  a  contingency  plan  for  mitigating 
the  identified  risks.  The  plan  should  establish 
risk  reduction  objectives  and  schedules,  as¬ 
sign  responsibility  and  priority  for  risk  reduc¬ 
tion  tasks,  and  develop  a  method  for  periodic 
reviews  and  assessments.  Various  manage¬ 
ment  techniques  to  reduce  risks  have  already 
been  mentioned  and  include: 

(a)  Rapid  prototyping; 

(b)  Incremental  development; 

(c)  Internal  (government)  program  reviews 
(at  least  monthly); 

(d)  Top  Ten  list  review; 

(e)  Early  demonstrations  and  testing  of  risk 
items; 

(f)  Government  inspections  and  audits  of  the 
software  development  process. 

Some  potential  problem  areas  include: 

(a)  Unrealistic  cost  and  schedule; 

(b)  Vague  or  incomplete  requirements; 

(c)  Inexperienced  developers; 

(d)  Inadequate  development  environment 
(tools  and  methodology). 

Identify  and  Track  Special  Interest  Items  - 

Special  interest  items  such  as  Government 
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Furnished  Items  (hardware,  software  and 
data)  and  subcontract  items  should  also  be 
tracked.  Any  items  delivered  to  the  contrac¬ 
tor  or  received  from  the  contractor  are  can¬ 
didates  for  tracking.  Certain  critical  internal 
deliveries  such  as  code  delivered  for  te.sting 
are  candidates  as  well.  The  same  approach 
used  for  tracking  the  risk  items  above  can  also 
be  used  to  track  special  interest  items. 

Requirements  Must  be  Testable  and  Trace¬ 
able  -  Requirements  must  be  testable  in  order 
to  validate  the  system  performance.  In  some 
cases,  actual  testing  may  be  impractical  due  to 
physical  constraints,  cost,  or  other  considera¬ 
tions.  When  this  is  the  case,  system  perfor¬ 
mance  must  be  validated  through  inference  or 
analysis  and  reflected  in  a  testability  matrix. 
Traceability  is  a  key  factor  throughout 
development  and  becomes  even  more  impor¬ 
tant  during  follow-on  support.  Requirements 
must  be  traceable  from  the  system  specifica¬ 
tion  down  through  design,  integration  testing 
and  DT&E.  Traceability  must  occur  in  both 
dlrections-results  of  a  test  report  must  track 
back  to  the  requirements.  The  only  effective 
way  to  handle  this  for  large  programs  is 
through  some  form  of  automation. 
Requirements’  traceability  should  be  an  in¬ 
tegral  part  of  the  contractor’s  configuration 
management  process. 

Use  Checklists  for  Design  Reviews  -  Without 
adequate  preparation  design  reviews  can  be¬ 
come  nothing  more  than  hectic  "dog  and  pony 
shows."  Design  Reviews  should  be  a  major 
part  of  the  software  quality  program.  Pro¬ 
gram  office  personnel  should  be  prepared  for 
a  review  by  arming  themselves  with  ap¬ 
propriate  analy.ses  and  a  checklist  of  impor¬ 
tant  and  critical  questions  and  adhering  as 
much,  as  possible,  to  an  agreed  to  formal 
agenda.  The  PM  must  do  his  homework 
through  verification  and  analyses  of  critical 
areas  that  support  the  design  approach.  MIL- 


STD-1521B,  Technical  Reviews  and  Audits  for 
Systems,  Equipments,  and  Computer  Software, 
provides  guidance  on  preparing  checklists  for 
various  reviews.  The  checklists  should  also 
include  special  interest  items  and  risk  areas. 

Formal  Reviews  Must  be  Viewed  as  Quality 
Gates  -  PMs  must  approach  the  formal  review 
as  a  checkpoint  for  determining  whether  or 
not  the  project  is  ready  to  proceed  to  the  next 
phase.  A  contractor  should  not  be  allowed  to 
complete  a  design  review  if  it  hasn’t  satisfied 
all  the  requirements  imposed  by  the  design 
review.  If  the  PM  decides  to  proceed  to  the 
next  phase  with  a  less  than  satisfactory  techni¬ 
cal  review,  because  of  political  or  schedule 
considerations,  the  risks  involved  must  be 
known  and  a  contingency  plan  developed. 
Too  often,  however,  the  risks  are  much  higher 
than  perceived.  It  is  essential  to  have  a  stable 
baseline  since  early  mistakes  become  much 
harder  and  co.stlier  (in  time  and  money)  to 
correct  when  they  are  discovered  late  in  the 
development  process.  Proceeding  before  one 
is  ready,  usually  increases  the  program  risk. 

Conduct  Periodic  Inspections  and  Audits  - 

Periodic  government  inspections  and  audits 
can  be  useful  when  applied  consistently. 
De.spite  the  fact  that  auditors  can  be  frequent¬ 
ly  wrong  in  their  assessments,  the  audit  is  an 
es,sential  tool  of  managements.  Whether  the 
audit  is  right  or  wrong,  the  very  process  of 
having  an  audit  impo.ses  discipline  on  the  or¬ 
ganization.  Ideas  on  organization,  design,  and 
process  are  interchanged.  Often  the  project  or 
program  improves  because  of  the  intensified 
attention  [4].  Inspections  generally  use  a 
checklist  to  determine  the  specification  and 
design  completeness.  Audits  are  similar  to 
inspections  with  the  additional  factor  of 
determining  requirements’  traceability. 
These  techniques  can  be  valuable  when  ap¬ 
plied  to  the  interfaces.  What  better  way  to 
integrate  the  system  than  to  ensure  that  the 
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hardware  and  software  properly  communi¬ 
cate  with  each  other?  In  addition  to  inspec¬ 
tions,  the  contractor  should  conduct 
walkthroughs  as  a  standard  business  practice 
[7].  When  conducting  inspections  and 
walkthroughs,  it  is  important  that  the  contrac¬ 
tor  establish  clear  entry  and  exit  criteria. 

Use  Statistics  Generated  by  Contractor  In¬ 
ternal  Reviews  -  The  contractor  must  have  a 
system  in  place  that  includes  a  method  for 
assessing  the  quality  and  progress  of  the  work. 
The  Software  Development  Plan  should  iden¬ 
tify  the  software  development  system  and  in¬ 
dicate  how  government  visibility  will  be 
provided.  This  is  accomplished  by  contrac¬ 
tually  requiring  the  developer  to  provide  the 
assessment  data  to  the  program  office.  The 
purpose  is  not  for  the  government  to  manage 
the  contractor’s  work  (that’s  the  contractor’s 
job)  but  to  communicate  program  develop¬ 
ment  status  and  product  quality.  The  contrac¬ 
tor  should  gather  statistics  from  the  internal 
walkthroughs  and  inspections  and  use  this 
data  to  manage  the  software  development. 
The  contractor’s  mechanism  for  software 
development  constitutes  the  process  control 
system.  A  software  process  control  system  is 
analogous  to  a  manufacturing  process  control. 

Integration  Must  be  Visible  on  M  ^ster 
Schedule  -  Integration  brings  the  interfaces 
together.  They  occur  at  all  levels:  software  to 
software,  software  to  hardware  and  software 
to  systems.  Integration  and  integration  testing 
must  be  planned  and  be  highly  visible  to  the 
government  PM.  The  development  of 
hardware  and  software  must  be  approached 
as  a  maturing  process  strategy.  Pieces  of  the 
system  should  be  brought  together  in  a 
planned,  logical  fashion.  Some  level  of  con¬ 
fidence  in  the  hardware  and  software  com¬ 
ponents  must  be  established  before 
proceeding  with  higher  level  integration.  If 
this  isn’t  done,  problems  that  occur  during 


integration  will  be  difficult  to  diagnose  since 
they  may  be  in  either  component— hardware 
or  software.  Integration  of  critical  system 
components  should  be  closely  tracked 
throughout  the  development  process. 

9.5  REQUIREMEN  k  S  /  PROTOTYPING 

The  PM  can  have  a  major  impact  on  the  suc¬ 
cess  of  the  program  during  the  requirements 
definition  phase.  Investment  of  time  and 
resources  in  requirements  definition  has  the 
biggest  schedule  and  cost  payoffs.  Clear  and 
unambiguous  requirements,  however,  seem 
to  be  an  elusive  ingredient  in  any  program. 
Since  software  development  is  an  intellectual 
process  that  must  be  captured  in  some  tan¬ 
gible  form,  how  does  one  approach  require¬ 
ments  definitions?  Two  tools  available  to  aid 
the  process  are  formal  specification  develop¬ 
ment  tools  and  rapid  prototyping. 

9.5.1  Specification  Development  Tools 

Specifications  are  generally  written  in  prose 
and  they  suffer  from  significant  interpretation 
and  traceability  problems.  They  are  also  very 
error  prone.  Formal  specification  develop¬ 
ment  tools  are  designed  to  alleviate  these 
problems.  Because  a  formal  specification 
language  is  much  more  specific  than  prose,  it 
can  be  automated.  This  provides  the  use  of 
computer  assistance  to  check  for  errors,  com¬ 
pleteness,  consistency,  and  traceability. 
Some  commonly  used  formal  specification 
development  tools  are: 

PSL/PSA:  Problem  Statement  Lan¬ 
guage/Problem  Statement  Analyzer.  This 
tool  was  originally  developed  for  data 
processing  applications.  It  is  widely  used  in 
other  applications  [8], 

RSL/REVS:  Requirements  Statement  Lan¬ 
guage/Requirements  Engineering  Validation 
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System.  This  is  a  real-time  process  control  beginning.  The  PM  should  incorporate  rapid 
tool  [9].  prototyping  as  part  of  the  contract. 


SADT:  Structured  Analysis  and  Design  Tech¬ 
nique.  This  tool  analyzes  the  interconnecting 
structure  of  any  large,  complex  systems.  It  is 
not  restricted  to  software  systems  [10]. 

SSA:  Structured  System  Analysis.  The  Gane 
and  Sarson  version  is  used  in  data  processing 
applications  that  have  database  require¬ 
ments.  The  DeMarco  version  is  suited  to  data 
flow  analysis  of  software  systems  [11,12]. 

Gist:  Textural  language  developed  at  USC/In- 
formation  Sciences  Institute.  This  tool  is  use¬ 
ful  for  developing  object-oriented 
specifications  and  designs.  It  is  a  refinement 
of  specifications  into  source  code  [13]. 

9.5.2  Rapid  Prototyping 

Rapid  prototyping  may  be  the  most  powerful 
tool  available  to  analyze  and  refine  require¬ 
ments  and  should  be  encouraged  as  a 
requirements’  definition  tool  from  the  very 


A  software  rapid  prototype  is  an  analytical 
tool  for  refining  software  requirements.  It  is 
used  during  the  requirements  analysis  phase 
with  minimal  constraints  on  choice  of 
programming  languages,  documentation,  and 
use  of  standards.  In  essence,  it  entails  the 
almost  unconstrained  development  of  a 
software  package  with  the  primary  goal  of 
achieving  quick  results. 

The  objective  is  to  compare  these  quick 
results  with  the  initial  system  requirements. 
By  allowing  the  user  a  quick  look  at  the  poten¬ 
tial  end  product,  they  will  be  better  able  to 
answer  the  questions  "Is  this  what  you 
wanted?"  or  "Is  this  what  you  meant?" 
Documentation  is  minimal  and  it  is  not 
deliverable.  Once  a  decision  is  made  to 
deliver,  it  ceases  to  be  a  rapid  prototype.  This 
is  analogous  to  the  hardware  requirements 
and  design  process  of  using  engineering 
models,  breadboards  and  brassboards.  This  is 
not  to  say  that  one  would  not  use  an  approach 
of  developing  software  using  a  "prototype"  (as 
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opposed  to  a  "rapid  prototype")  for  delivery 
and  operation.  This  is  yet  another  method 
called  evolutionary  development  and  it  will  be 
discussed  later  in  this  chapter.  The  value  of  a 
rapid  prototype  is  in  the  capability  to  better 
communicate  software  requirements  during 
the  requirements  analysis  process.  What  bet¬ 
ter  way  to  communicate  with  the  users  than  to 
present  them  with  an  artifact  that  represents 
the  system?  Often  the  rapid  prototype  will 
represent  the  user  interface  (controls  and  dis¬ 
plays)  providing  for  both  input  and  output.  It 
may  execute  representative  scenarios  or 
operational  profiles  to  determine  the  validity 
of  the  specifications.  The  prototype  can  be 
used  throughout  development,  including  Full 
Scale  Development,  to  test  out  design  con¬ 
cepts  as  well  as  develop  a  priori  test  results  for 
future  testing  of  the  actual  product.  One  must 
remember  that  a  rapid  prototype  does  luil 
evolve  into  a  deliverable  software  product.  It 
should  be  discarded  once  it  has  been  used  to 
revise  and  refine  the  requirements. 

9.5.3  Incremental  and  Evolutionary 
Development 

Incremental  and  evolutionary  development 
are  techniques  for  dealing  with  large,  complex 
systems.  Figure  9-2  illustrates  incremental 
development.  Design  begins  after  the  system 
and  software  requirements  have  been 
baselined. 

The  objective  of  incremental  development  is 
to  produce  a  complex  software  product  by 
building  the  total  .system  capability  in  ever 
increasing  increments.  The  second  software 
delivery  or  increment  will  have  more 
capability  than  the  first  delivery,  the  third 
more  than  the  second  and  so  on.  As  the  saying 
goes  "If  you  have  to  eat  an  elephant,  eat  him 
one  bite  at  a  time."  Software  development  for 
Inter-Continental  Ballistic  Missiles  (ICBMs), 
for  example,  is  developed  this  way.  The 


Operational  Flight  Program  (OFP)  may  be 
designed  and  developed  in  three  increments. 

This  incremental  approach  must  be  pre¬ 
planned  with  the  overall  development 
strategy  and  test  plan.  The  first  system  level 
capability  to  be  demonstrated  is  the  capability 
to  fly  from  point  "A"  to  point  "B".  This  re¬ 
quires  the  appropriate  planning  for  develop¬ 
ing  and  integrating  both  hardware  and 
software  components  in  support  of  the 
scheduled  events.  The  first  increment  would 
have  the  basic  flight  control,  navigation, 
guidance,  and  task  control  functions.  The 
next  increment  would  add  the  system 
capability  for  multiple  re-entry  vehicle 
deployment.  Again,  meticulous  planning 
would  have  preceded  this  activity  to  make 
sure  that  the  appropriate  hardware  and 
software  components  are  designed  and 
developed  on  an  integrated  schedule.  For  the 
software,  this  second  increment  or  "build" 
would  have  the  same  capabilities  as  the  first 
increment  plus  the  capability  to  deploy  multi¬ 
ple  re-entry  vehicles  during  a  flight.  The  final 
software  increment  may  contain  maintenance 
diagnostic  capability  and  continuous  naviga¬ 
tion  instrument  calibration  capability.  One 
begins  with  a  minimal  program  and  then  adds 
additional  capability  until  the  complete  sys¬ 
tem  is  developed. 

Evolutionary  development  is  very  similar  to 
incremental  development  but  it  is  more  long 
term.  While  incremental  development  oc¬ 
curs  during  a  single  development  phase,  typi¬ 
cally  during  FSD,  evolutionary  development 
can  occur  over  several  phases  or  be  part  of  a 
pre-planned  product  improvement  approach. 
Evolutionary  development  is  the  recom¬ 
mended  approach  for  large  Command,  Con¬ 
trol,  Communications  and  Intelligence  (C^I) 
systems.  Rapid  prototyping  can  be  combined 
with  this  approach  to  help  define  the  require¬ 
ments.  The  evolutionary  approach  helps  to 
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deal  with  "fuzzy"  requirements  where  the 
general  requirements  are  known  but  the 
details  are  lacking.  For  example,  in  a  large 
C^I  program  the  PM  may  know  that  he  must 
design  a  system  to  communicate  with  multiple 
users,  be  able  to  react  to  rapidly  changing 
threats,  and  be  able  to  adapt  to  the  environ¬ 
ment  in  real-time.  He  may  not,  however, 
know  the  requirements  of  all  the  users,  who 
may  in  turn  not  know  themselves  until  they 
are  able  to  work  with  the  actual  equipment  in 
a  scenario  or  simulated  environment.  The 
evolutionary  development  approach  would 
develop  an  early  model  with  flexibility  and 
growth  specifically  as  part  of  its  design.  This 
model  would  then  be  used  under  actual  or 
simulated  conditions  to  provide  feedback  to 
update  the  requirements.  This  approach 
could  continue  indefinitely. 

Plan 

-  Cost  and  schedule 

-  Development  and  support 
Select  tangible  inchstones 

Review  schedule  after  reciuircincnts  are  defined 
Scrub  requirements 
Build  rapid  prototypes 

Create/Update  software  size  and  cost  estimates 
Design  within  the  system’s  constraints 
Don’t  let  hardware  needlessly  constrain  software 
delopment 

Establish  CRWCJ  and  provide  software  support 
Beware  of  government  furnished  products  and 
subcontracts 

Strongly  consider  incremental  development  for 
large  systems 

Create  and  manage  schedule,  memory, 
throughput  margins 

Understand  the  contractor’s  development 
process 

Establish  an  internal  control  system 

Use  software  metrics 

Select  resources  and  language 

Fig.  9-3  Program  Management  Checklist 


9.6  SUMMARY 

The  problems  associated  with  managing 
software  development  can  be  overwhelming. 
But  one  must  never  forget  that  software 
development  is  manageable.  This  chapter 
has  discussed  some  of  the  tools  available  to 
the  program  manager  to  help  in  planning  a 
development  strategy  as  well  as  assisting  in 
the  daily  management  of  the  program.  The 
Program  Management  Checklist  in  Figure  9-3 
serves  as  an  additional  reminder  of  the  key 
elements  that  have  been  discussed 
throughout  this  text. 
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CHAPTER  10 


SOFTWARE  CONFIGURATION 
MANAGEMENT 


10.1  INTRODUCTION 

The  dictionary  defines  configuration  as  the 
"relative  disposition  of  the  parts  or  elements 
of  an  item"  and  defines  management  as  the 
"act  or  manner  of  handling,  directing,  or  con¬ 
trolling"  [1].  Configuration  management 
(CM),  then,  can  be  generally  defined  as  the 
act  of  controlling  all  elements  of  a  particular 
item.  When  applied  to  weapon  systems,  CM 
is  the  system  engineering  management 
process  that:  creates  the  system  components, 
identifies  the  functional  and  physical  charac¬ 
teristics  of  those  components,  controls  chan¬ 
ges  to  those  characteristics,  and  records  the 
status  of  any  changes  implemented.  It  is  the 
means  through  which  the  integrity  and  con¬ 
tinuity  of  design,  engineering,  and  cost 
tradeoff  decisions  made  between  technical 
performance,  producibility,  operability,  and 
supportability  are  reported,  communicated, 
and  controlled  [2]. 

Software  configuration  management  (SCM) 
is  formally  defined  as  the  process  used  to 


identify  the  software  configuration  com¬ 
ponents  of  a  system  for  the  purpose  of  sys¬ 
tematically  controlling  changes  to  the 
configuration  and  maintaining  the  integrity 
and  traceability  of  that  configuration 
throughout  the  system  life  cycle.  This  chapter 
explains  the  basic  elements  of  SCM  and  shows 
how  the  application  of  these  elements  to  the 
software  development  cycle  facilitates  the 
transformation  of  software  into  a  visible  and 
manageable  entity  [3].  Figure  10-1  illustrates 
the  interrelationship  of  the  four  functions  of 
SCM;  identification,  control,  status  account¬ 
ing,  and  audits  [4]. 

The  developer  (a  contractor  or  another 
government  agency)  and  the  government 
procuring  agency  both  apply  CM  procedures 
to  a  specific  development  program.  This 
chapter  concentrates  on  CM  practices  as 
described  in  DOD-STD-48()A,  Configuration 
Control,  Engineering  Changes,  Deviations,  and 
Waivers,  M1L-STD-483A,  Configuration 
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Management  Practices  for  Systems,  Equip¬ 
ment,  Munitions,  and  Computer  Resources  and 
DOD-STD-2167A,  Defense  System  Software 
Development. 

10.2  CONFIGURATION  IDENTIFICA¬ 
TION 

Configuration  identification  determines  how 
to  divide  the  software  system  for  ease  in 
managing  and  controlling  change.  A  system 
comprises  a  physical  and  a  functional  con- 


Software  configuration  identification  means 
specifying  and  identifying  all  system  software 
components  throughout  its  life  cycle,  from  the 
development  of  specifications  to  the  genera¬ 
tion  of  actual  code  [4].  The  definition  of 
Computer  Software  Configuration  Items 
(CSCIs),  along  with  their  functional  and 
physical  characteristics,  normally  occurs 
during  the  Demonstration  and  Validation 
(D/V)  phase  of  the  acquisition  life  cycle  and 
prior  to  the  System  Design  Review  (SDR). 
MIL-STD-483A,  Appendix  XVII,  provides 


CONFIGURATION  MANAGEMENT 


Fig.  10-1  Configuration  Management  Functions 


figuration.  Physical  Configuration  refers  to 
the  detailed  design  or  physical  attributes  and 
it  is  normally  described  by  hardware  drawings 
and  software  code  listings.  Functional  Con¬ 
figuration  refers  to  the  functions  a  system  or 
unit  performs  and  it  is  primarily  established 
by  hardware  and  software  requirements’ 
documents  [3].  The  functional  description  of 
software  is  documented  in  interface  specifica¬ 
tions,  product  description,  test  procedure  and 
detailed  design  specifications. 


the  following  guidance  for  selecting  con¬ 
figuration  items  (CIs): 

(a)  Select  CIs  based  on  functional  and  perfor¬ 
mance  parameters  which  must  be  controlled 
to  satisfy  an  overall  end  use  function  (e.g., 
defensive  avionics  system  software); 

(b)  Select  CIs  which  require  an  optimum  level 
of  government  control  during  acquisition 
(e.g.,  code  verification  software); 
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(c)  Select  CIs  based  on  the  need  to  control 
their  inherent  characteristics  or  to  control 
their  interface  with  other  CIs  (e.g.,  controls 
and  displays  software); 

(d)  When  selecting  CIs,  evaluate  other  factors 
such  as  schedule,  the  engineering  release  sys¬ 
tem,  financial  impact,  and  new,  modified,  or 
existing  design  parameters. 

The  process  of  configuration  identification 
provides  a  way  to  isolate  system  components 
to  control  their  development.  There  are  four 
steps  to  software  configuration  identification. 
First,  the  software  system  is  broken  down  into 
a  number  of  known  manageable  parts  or 
CSCIs.  Second,  these  CSCIs  are  uniquely 
named.  Third,  as  these  parts  change  with  time, 
the  various  versions  that  appear  are  uniquely 
identified.  Fourth  a  change  control  process  is 
instituted  to  ensure  knowledge  and  control  of 
all  changes  that  occur  throughout  the  lifecycle 
of  the  program.  The  first  step  is  closely  as¬ 
sociated  with  the  processes  of  specification, 
analysis,  and  design.  Steps  two  and  three  re¬ 
quire  rigorous  enforcement  of  step  four, 
standards  and  procedures.  Figure  10-2  repre¬ 
sents  a  generic  breakdown  of  a  software  sys¬ 
tem  into  various  CSCIs,  computer  software 
components  (CSCs),  and  computer  software 
units  (CSUs).  The  contractor  will  normally 
propose  a  list  of  CSCIs  based  on  the  Request 
For  Proposal  (RFP)  Work  Breakdown  Struc¬ 
ture  (WBS),  the  design  requirements, 
management  structure,  and  available  resour¬ 
ces.  This  structure  of  CSCIs  should  be 
reflected  in  the  contractor’s  Software 
Development  Plan  (SDP)  and  Configuration 
Management  Plan  (CMP).  The  contractor’s 
CM  methodology  is  documented  in  the  CMP. 
The  CMP  and  SDP  are  normally  deliverables 
on  software  programs.  They  are  presented  for 
approval  to  the  government  in  accordance 
with  the  provisions  of  the  Contract  Data  Re¬ 
quirements  List  (CDRL). 


Care  must  be  taken  when  selecting  the  num¬ 
ber  of  configuration  items.  Too  many  con¬ 
figuration  items  may  increase  the 
management  and  administrative  efforts  re¬ 
quired  to  adequately  track  and  control  the 
status  of  the  CSCIs.  This  additional  effort 
may  delay  the  schedule  and  increase  the  cost 
of  the  software  development.  On  the  other 
hand,  too  few  CSCIs  may  minimize  the  pro¬ 
gram  office’s  visibility  into  the  software 
development  process,  tend  to  reduce  control 
of  the  software  design  and  possibly  lead  to 
operational  deficiencies. 

While  the  process  of  Cl  decomposition  allows 
for  the  management  of  small  units,  one  should 
remember  that  software,  like  hardware,  is  a 
sum  of  its  parts  and  must  be  managed  from  a 
system  approach.  CM  is  the  primary  respon¬ 
sibility  of  the  program  office.  Unfortunately, 
in  many  program  offices  the  CM  function  be¬ 
comes  a  low  priority  task  managed  by  a 
caretaker  group  and  is  not  considered  an  in¬ 
tegral  participant  in  the  daily  management  of 
the  program.  Because  of  this,  the  Program 
Manager  (PM),  as  well  as  other  functional 
managers,  may  inadvertently  affect  the  con¬ 
figuration  of  the  system  in  their  routine  inter¬ 
faces  with  the  developer.  There  should  be  a 
strong  relationship  between  the  CM  and 
other  functional  areas  in  the  program  office. 

10.3  CONFIGURATION  CONTROL 

Once  the  system  configuration  is  established, 
the  next  step  is  to  establish  a  method  to 
manage  and  control  changes.  Unlike 
hardware,  software  is  an  intangible  product 
difficult  to  "see"  and  more  difficult  to  manage. 
Software  can  still  be  properly  managed,  how¬ 
ever,  by  applying  configuration  control 
methods  to  the  development  process  and  on 
the  products  of  that  process.  Pudward  Bersoff 
defines  software  configuration  control  as 
"...the  orchestration  of  the  processes  by  which 
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Fig.  10-2  Breakdown  of  Softwai  C  System 

the  software  portion  of  a  system  can  achieve  software  Engineering  Change  Proposals 

and  maintain  visibility  throughout  its  journey  (ECPs),  test  asset  control,  test  software  con- 

through  the  life  cycle.  It  provides  the  tools  trol,  physical  test  item  control,  integration 

(i.e.,  documentation,  procedures,  and  an  or-  control,  and  version  control, 

ganizational  body)  to  control  the  system  im¬ 
plementation  as  well  as  changes  to  it"  [4].  To  understand  the  configuration  control 

process,  one  has  to  understand  the  differences 
The  configuration  and  change  control  process  between  Class  I  and  Class  II  changes.  Class  I 

includes:  interface  control,  baseline  manage-  changes  effect  form,  fit,  or  function;  although 

ment,  Configuration  Control  Boards  (CCBs),  other  factors,  such  as  cost  or  schedule,  can 

Software  Configuration  Review  Boards  cause  a  Class  I  change.  All  Class  I  changes 

(SCRB),  Software  Problem  Reporting  (SPR),  must  be  submitted  to  the  Government  CCB 
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for  approval.  All  other  changes  are  Class  II 
changes.  Class  II  changes  do  not  change  the 
scope  of  requirements,  do  not  effect  cost,  or 
change  the  contracted  schedule.  Examples  of 
Class  II  ciianges  are  editorial  changes  in 
documentation  or  minor  changes  which  don’t 
affect  the  established  baselines  or  the  func¬ 
tional  allocation  of  the  CSCI. 

For  government  approved  Class  I,  the  govern¬ 
ment  must  negotiate  with  the  contractor 
before  the  change  can  be  implemented.  A 
contractor  may  proceed  to  implement  a 
proposed  Class  I  change  before  government 
approval,  but  he  proceeds  at  his  own  risk. 

Class  II  changes  only  require  government 
concurrence  on  the  classification  before  the 
contractor  may  proceed  with  a  change.  This 
can  be  accomplished  by  government  plant 
representatives.  The  PM  should  ensure  that 
the  contractor  supplies  the  program  office 
with  copies  of  all  Cla.ss  I  and  Class  II  Draft 
changes  as  well  as  any  Class  II  changes  that 
have  been  approved  by  local  authorities. 
MIL-STD-483A,  Appendix  XIV  discusses 
software  Class  I  and  II  changes. 

10.3.1  Interface  Control 

An  interface  is  the  functional  or  physical  char¬ 
acteristics  which  serve  as  a  common  boundary 
between  two  or  more  items.  In  system 
development,  these  boundaries  are  found  be¬ 
tween  hardware  and  software,  hardware  and 
hardware,  operational  communication  stand¬ 
ards  and  the  system,  and  software  and 
software  configuration  items.  The  interfaces 
are  defined  by  electro-mechanical  charac¬ 
teristics,  reliability  and  maintainability  re¬ 
quirements  and  software  format,  timing,  and 
programming  language’s  requirements. 

The  program  office  controls  all  external  inter¬ 
faces  that  effect  the  system  under  develop¬ 


ment  through  groups  such  a  CCB,  an  Interface 
Control  Working  Group  (ICWG),  a  Test 
Planning  Working  Group,  and  a  Computer 
Resources  Working  Group  (CRWG).  The 
contractor  controls  all  internal  interfaces  to 
the  program  under  development  through  the 
contractor’s  own  configuration  control 
process  and  change  management  process. 

The  government  ICWG  concerns  itself  with 
issues  external  to  the  product  such  as  inter¬ 
operability,  logistics,  and  operational  issues. 
The  ICWG  is  usually  chaired  by  the  procuring 
agency’s  engineering  representative  and  is 
comprised  of  representatives  from  the 
development  organizations,  user  organiza¬ 
tions,  and  software  support  activities  as  shown 
in  Figure  10-3.  The  ICWG,  in  coordination 
with  the  CRWG,  define  and  control  current 
and  proposed  software  and  hardware  inter¬ 
faces,  obtain  and  assess  quantitative  interface 
data,  and  investigate  system  and  subsystem 
inter-operability  requirements. 

During  Full  Scale  Development  (FSD)  many 
programs  delegate  the  responsibility  for 
product  interface  control  to  the  development 
contractor.  As  the  system  matures  and 
proceeds  toward  test  and  delivery,  the  govern- 
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ment  gradually  assumes  more  of  this  respon¬ 
sibility.  The  contractor’s  ICWG  is  usually 
chaired  by  an  individual  from  the  engineering 
organization.  Associate  contractors  are  mem- 
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bers  and  the  responsible  DOD  agency  may  be 
an  observer.  The  contractor’s  ICWG  docu¬ 
ments  agreen.^nts  on  CSCI  interfaces  and 
hardware  configuration  item  (HWCIs)  inter¬ 
faces  involving  more  than  one  contractor. 
The  contractor’s  ICWG  should  review  all  en¬ 
gineering  and  interface  changes  before  sub¬ 
mitting  then  to  the  Program  Office  [5]. 

10.3.2  Baseline  Management 

Baseline  management  is  the  process  of 
managing  the  derivation  of  the  contractual 
requirements  into  implementable  functions 
and  the  integration  of  those  functions  into  a 
system  that  demonstrates  compliance  with 
these  requirements. 

Within  each  of  the  development  phases, 
specific  reviews  (SRR,  SDR,  PDR,  CDR) 
have  been  established  to  examine  the 
contractor’s  compliance  with  the  require¬ 
ments  of  the  contract  and  his  progress  toward 
delivery  of  a  system.  This  is  accomplished 
through  documentation  such  as  specifica¬ 
tions,  design  documentation,  management 
reports,  and  test  reports.  At  the  completion 
of  each  review  these  documents  are  approved 


and  a  contractor’s  baseline  is  incremented  to 
indicate  government  approval  of  the 
contractor’s  design  and  progress  at  that  par¬ 
ticular  stage.  Documented  deficiencies  will 
indicate  the  contractor’s  progress  toward 
meeting  the  requirements’  baseline.  Tradi¬ 
tionally  there  have  been  three  baselines,  the 
functional,  allocated,  and  product  baselines. 
The  provisions  of  most  contracts  require  that, 
as  baselines  change,  the  system  integrity  be 
maintained.  Each  baseline  indicates  a  state  of 
design  that  becomes  progressively  more 
restrictive  and  finally  represents  the  actual 
hardware  and  software  developed. 

A  functional  baseline  is  typically  established 
at  the  completion  of  the  System  Design 
Review  (SDR)  and  approval  of  the  Type  A 
system  specification.  Formal  configuration 
control  for  the  .system  is  initiated  once  the 
system  specification  has  been  approved.  The 
allocated  baseline  for  the  system  is  estab¬ 
lished  once  the  allocated  baseline  for  each  Cl 
is  determined  and  the  Software  Require¬ 
ments  Specification  (SRS)  and  Interface  Re¬ 
quirements  Specifications  (IRS)  for  each 
configuration  item  have  been  approved.  For 
hardware,  this  normally  occurs  at  the  Prelimi- 
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nary  Design  Review  (PDR).  For  software,  control  changes  to  the  system.  Figures  10-4 

the  allocated  baseline  is  normally  established  and  10-5  illustrate  typical  membership  of 

at  the  completion  of  the  Software  Specifica-  government  and  contractor  CCBs.  The  com- 

tion  Review  (SSR)  but  no  later  then  the  CDR.  position  of  both  CCBs  are  very  similar;  how- 

The  product  baseline  is  typically  established  ever,  the  responsibility  of  the  government 

at  the  completion  of  the  Physical  Configura-  CCB  is  at  the  system  level.  The  contractor  has 

tion  Audit  (PCA)  after  the  Type  C  specifica-  additional  configuration  control  respon- 

tions  are  approved.  When  programmatic  sibilities  at  lower  hierarchical  levels  for  both 

issues  preclude  the  PM  from  formally  estab-  software  and  hardware  as  depicted  in  Figure 

lishing  a  baseline,  alternatives  include  an  in-  10-6  [3]. 

formal  or  developmental  baseline  agreement 

between  the  contractor  and  program  office.  103.4  Software  Configuration  Review  Board 

Informal  baselines  imply  a  sharing  of  con-  A  Software  Configuration  Review  Board 

figuration  control  responsibility  between  the  (SCRB)  reviews  and  evaluates  all  proposed 

contractor  and  the  program  office.  It  allows  changes  to  the  software  baselines  and  deter- 

the  contractor  a  large  degree  of  latitude  in  mines  the  processing  and  disposition  of 
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processing  changes  to  the  baseline  in  pursuit 
of  the  correct  design  and  relieves  the  govern¬ 
ment  of  the  formal  structured  change  control 
process.  However,  the  contractor  proceeds 
on  risk  and  is  not  encouraged  to  formalize  the 
changes  it  has  initiated. 

10.3.3  Configuration  Control  Board 

The  Configuration  Control  Board  is  the  or¬ 
ganizational  body  within  the  development  or¬ 
ganization  responsible  for  formal  processing 
of  changes  to  established  baselines.  The  func¬ 
tion  of  the  CCB  is  to  approve,  monitor,  and 


software  problem  reports.  The  SCRB  serves 
as  a  filter  for  the  CCB  on  software  related 
matters.  Software  problem  report.s,  incident 
reports,  change  requests,  and  change 
proposals  are  first  submitted  to  the  SCRB  for 
review  and  evaluation.  The  SCRB  deter¬ 
mines  if  the  pending  software  changes  should 
be  disapproved  or  forwarded  to  the  CCB  for 
formal  approval  as  baseline  changes.  Prior  to 
its  decision,  the  SCRB  would  have  reviewed  a 
number  of  proposed  software  changes, 
prioritized  their  urgency,  and  determined  if 
the  changes  should  be  made  singly  or  in  com¬ 
patible  groupings  or  block  changes. 
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As  with  the  Configuration  Control  Board,  the 
SCRB  is  also  used  by  both  the  government 
and  the  contractor.  Figure  10-7  illustrates  the 
typical  composition  of  these  two  SCRBs. 

In  the  government,  the  SCRB  normally 
begins  to  functions  as  the  government  as¬ 
sumes  management  responsibility  of  the 
baselines  and  continues  during  the  produc¬ 
tion/support  phases  of  the  acquisition  cycle  of 
a  weapon  system.  That  is  one  of  the  reasons 
for  including  the  Software  Support  Activity  as 
part  of  its  membership.  During  the  transition 
to  organic  support,  the  program  manager  may 
choose  to  have  the  contractor  as  a  non-voting 
member  of  the  SCRB.  As  a  minimum,  the 
SCRB  should  be  conducted  annually  in  con¬ 
junction  with  planned  upgrades  (or  releases) 
of  the  software  system.  However,  the  SCRB 
may  be  convened  at  any  time  for  major  (criti¬ 
cal)  deficiencies  discovered  in  the  software. 

The  software  developer  typically  employs  an 
internal  SCRB  during  development.  Since 
the  SCRB  is  a  software  board,  its  membership 
is  made  up  of  mostly  software  oriented  repre¬ 
sentatives.  The  software  manager  chairs  the 


board  and  has  final  approval  authority.  The 
SCRB  should  meet  routinely  to  act  on  all 
submitted  software  problem  reports  (SPRs). 
The  actions  of  the  SCRB  may  require  the 
contractor  to  submit  ECPs  and  other  contract 
modifying  requests  in  order  to  implement  a 
solution  to  a  problem.  Normally  if  a  change 
does  not  effect  the  baseline,  it  is  implemented 
on  approval  by  the  contractors  CCB.  Any 
change  that  affects  a  program  baseline  must 
be  transmitted  to  the  contractors  CCB  for 
disposition  [6].  Disposition  may  take  the  form 
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of  internal  direction  to  proceed  because  the 
change  does  not  impact  the  design  baseline  or 
a  transmittal  to  the  Government  Program  Of¬ 
fice  for  formal  government  CCB  and  follow- 
on  contracting  action. 

The  Software  Development  Library  (SDL) 
librarian  is  a  key  member  of  the  developer’s 
SCRB  and  acts  as  the  board  recorder.  Since 
the  SCRB  is  administered  and  supported 
through  the  SDL,  the  SDL  librarian  is  re.spon- 
sible  for  tracking  the  status  of  the  Software 
Problem  Reports  (SPRs),  forwarding  the 
recommended  SPRs  to  the  contractor’s  CCB 
for  disposition,  incorporating  the  imple¬ 
mented  changes  in  the  SDL,  and  maintaining 
up'to-date  records  of  these  reports. 

10J.5  Configuration  Control  Process 

The  configuration  control  process  (Figure  10- 
8)  is  d  very  time  consuming  and  active  process. 
It  is  the  heart  of  the  configuration  control 
function.  Both  the  government  and  the 
developer  follow  a  similar  internal  change 
control  process  which  results  in  software 
ECPs  forwarded  to  the  government  CCB  for 
evaluation  and  approval. 

The  configuration  control  process  begins  with 
the  initialization  of  a  change  to  an  established 
baseline  (e.g.,  allocated,  functionai,  pro¬ 
gram).  These  changes  may  be  initiated  by 
government  direction,  ICWG  activity,  or  con¬ 
tractor/subcontractor  activity.  If  the  change  is 
initiated  by  the  government,  the  developer’s 
engineering  activity,  in  coordination  with  its 
ICWG,  will  analyze  the  technical,  cost,  and 
schedule  impacts  of  the  proposed  change. 
Once  a  change  has  been  reviewed  and 
evaluated,  the  developer  will  categorize  it  as 
either  Class  I  or  II.  All  software  related  issues 
will  be  forwarded  to  the  SCRB  for  review, 
evaluation,  and  disposition.  The  contractor  is 
asked  to  cost  this  change  and  participates  in  a 


contract  modification.  If  the  contractor  in¬ 
itiates  the  change  request,  the  process  is 
similar.  The  government  CCB  dispositions 
the  change  request  and  formally  notifies  the 
contractor  by  PCO  letter.  If  it’s  an  approved 
Class  I  change,  the  contractor  and  the  govern¬ 
ment  negotiate  the  cost  and  schedule  impact. 

Class  I  changes  follow  a  more  formal  route 
through  the  CCB.  The  developer  will  submit 
all  proposed  Class  I  changes  to  its  CCB  for 
review  and  disposition.  The  CCB  will  deter¬ 
mine  if  the  change  should  be  formally  sub¬ 
mitted  to  the  government.  If  they  decide  not 
to,  they  may  choose  to  follow  an  informal 
route  by  submitting  preliminary  documenta¬ 
tion  consisting  of  an  Advanced  Change/Study 
Notice,  an  Engineering  Change  Request,  or  a 
preliminary  ECP.  If  this  i.:Tormal  documenta¬ 
tion  is  approved,  the  government  will  notify 
the  developer  to  submit  formal  documenta¬ 
tion.  The  formal  documentation  may  consist 
of  an  ECP,  a  Specification  Change  Notice 
with  specification  page  changes,  a  Request  for 
Deviation/Waiver,  an  Interface  Revision 
Notice,  and  supporting  cost  data.  Once  ap¬ 
proved/disapproved,  the  program  office  will 
notify  the  contractor  and  both  will  monitor 
the  implementation  status. 

When  Class  II  changes  are  submitted  to  the 
government,  the  government’s  Defense  Plant 
Representative  Office  (DPRO)  may  approve 
the  classification  and  sign  off  on  the  change. 
Once  the  change  is  approved,  the  contractor 
may  implement  and  monitor  the  status  of  the 
change.  Class  II  changes  which  do  not  receive 
government  concurrence  may  be  submitted  as 
Class  1  changes. 

10.4  CONFIGURATION  STATUS  AC¬ 
COUNTING 

Configuration  status  accounting  is  the  mana¬ 
gement  information  system  that  provides 
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Fig,  10-8  Change  Control  Process 


traceability  of  changes  to  configuration 
baselines  and  facilitates  the  effective  im¬ 
plementation  of  changes.  It  consists  of  reports 
and  records  documenting  CSCI  change  ac¬ 
tions.  The  basic  documentation  includes  the 
Configuration  Identification  Index  and  Status 
Accounting  Report.  DOD-STD-2167A 
provides  specific  guidance  for  the  status  ac¬ 
counting  of  software,  MIL-STD-482A  stand¬ 
ardizes  data  elements  with  regard  to  format, 
frequency,  and  record  keeping  [2]. 

10.4.1  Software  Development  Library 

The  contractor’s  Software  Development 
Library  (SDL)  is  defined  in  DOD-STD- 
2167A  as  a  controlled  collection  of  software, 
documentation,  and  associated  tools  and  pro¬ 
cedures  used  to  facilitate  the  orderly  develop¬ 


ment  and  subsequent  support  of  software.  An 
SDL  provides  storage  of,  and  controlled  ac¬ 
cess  to,  software  and  documentation  in 
human-readable  form,  machine-readable 
form,  or  both.  It  also  ensures  compliance  with 
process  procedures.  It  serves  as  the 
contractor’s  single  point  of  configuration  con¬ 
trol  during  all  phases  of  a  contract.  The  SDL 
maintains  established  project  baselines,  and 
monitors  and  controls  the  project  develop¬ 
ment  configuration  baselines  and  data 
products.  Software  products  consist  of 
documentation  and  listings,  source  code,  ex¬ 
ecutable  (machine)  code,  and  status  records. 
The  SDL  responsibilities  consist  of; 

(a)  Establishing  technical  configuration  con¬ 
trol  and  monitoring  the  quality  of  software 
products; 
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(b)  Maintaining  organizational  facilities  for 
baselining  and  controlling  the  content  of 
software  products; 

(c)  Establishing  reportii.g  procedures  for 
resolving  software  design  or  implementation 
issues  and  documenting  the  contents  of  the 
library’s  data  products  [5]. 

(d)  Serving  as  the  repository  for  test  software 
and  maintaining  configuration  control  of 
software  testing. 

The  SDL  librarian  stores  the  completed 
software  products.  This  includes  listings  of 
tested  functions,  development  folders  as¬ 
sociated  with  the  function,  and  functional  test 
results.  He  or  she  also  maintains  current  list¬ 
ings  and  copies  of  products  under  develop¬ 
ment,  updates  them  as  required,  tracks  the 
functions  to  produce  a  historical  development 
record,  and  controls  the  distribution  of  copies 
to  appropriate  personnel.  The  librarian  en¬ 
sures  compliance  with  contractual  design 
guidelines  and  maintains  completed  software 
development  folders.  As  recorder  for  the 
SCRB,  the  SDL  librarian  schedules  and  coor¬ 
dinates  the  SCRB  meetings  and  reviews  the 
SPRs  to  insure  that  these  reports  are  ready  for 
discussion  at  the  SCRB.  The  librarian  tracks, 
monitors,  and  records  the  status  of  action 
items  assigned  to  the  SPRs,  and  prepares  and 
distributes  the  minutes  of  SCRB  meetings. 

10.4.2  Software  Development  Folder 

A  Software  Development  Folder  (SDF)  is 
defined  in  DOD-STD-2167A  as  a  repository 
for  a  collection  of  material  pertinent  to  the 
development  configuration.  The  contents  of 
the  SDF  typically  include  (either  directly  or 
by  reference)  design  considerations  and  con¬ 
straints,  design  documentation  and  data, 
schedules  and  status  information,  test  re¬ 
quirements,  test  cases,  test  procedures,  and 


test  results.  The  contractor  documents  the 
development  of  each  CSU,  CSC,  and  CSCI  in 
SDFs.  The  SDFs  constitute  the  first  line  of 
configuration  management.  The  Configura¬ 
tion  Manager  is  responsible  for  their  content 
and  completeness  although  this  responsibility 
is  typically  delegated  to  the  Lead  Programmer 
or  the  project  leader  of  a  particular  software 
effort.  The  role  of  CM  is  to  ensure  that  the 
SDFs  are  complete  before  they  are  formally 
accepted  into  the  SDL  and  baselined.  This 
step  is  crucial.  Many  lower  level  functions  are 
tested  and  validated  through  the  use  of  the 
SDFs.  The  SDFs  are  maintained  for  the  dura¬ 
tion  of  the  contract  and  will  be  made  available 
for  government  review  upon  request. 

10.5  CONFIGURATION  AUDITS 

The  fourth  function  of  government  configura¬ 
tion  management  is  to  perform  a  set  of  con¬ 
figuration  audits  to  verify  that  the  selected 
configuration  items  conform  to  the  specifica¬ 
tions  and  related  technical  data.  There  are 
two  types  of  audits  conducted:  a  Functional 
Configuration  Audit  (FCA)  and  a  Physical 
Configuration  Audit  (PCA).  The  details  of 
these  audits  are  described  in  MIL-STD- 
1521 B,  Technical  Reviews  and  Audits  for  Sys~ 
terns,  Equipments,  and  Computer  Software. 

The  software  FCA  is  a  formal  examination  of 
the  functional  characteristics  of  a  CSCI  prior 
to  acceptance  to  verify  that  the  CSCI  has 
achieved  the  performance  specified  in  its 
Software  Requirements  Specification  (SRS) 
and  Interface  Requirements  Specification 
(IRS).  Other  technical  documentation  such 
as  the  Software  Test  Plan  (STP),  the  Software 
Test  Descriptions  (STD),  the  Software  Test 
Reports  (STR),  and  minutes  of  the  design 
reviews  are  evaluated  for  completeness.  The 
FCA  validates  the  development  of  a  CSCI  has 
been  satisfactorily  completed  and  that  the 
CSCI  performs  as  defined  in  the  contract. 
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The  PCA  is  a  formal  examination  of  the  "as- 
built"  version  of  the  CSCI  as  described  in  the 
Software  Product  Specification  (SPS).  The 
source  code  for  each  CSCI  is  compared  with 
the  associated  documentation  (SRS,  IRS, 
software  design  documents,  Interface  Design 
Documents,  and  Version  Description  Docu¬ 
ments)  for  accuracy  and  completeness.  The 
PCA  establishes  the  product  baseline  for  each 
CSCI  and  may  occur  prior  to  hardware  and 
software  system  integration  and  testing. 
However,  system  level  PCAs  are  often 
delayed  until  after  system  integration  and 
testing  is  completed.  The  PCA  also  verifies 
that  the  tested  object  code  can  be  recreated 
or  compiled  from  the  baselined  source  code. 

A  word  of  caution  should  be  noted  in  per¬ 
forming  FCA/PCAs.  In  a  highly  integrated 
hardware  and  software  effort,  it  is  normally 
impossible  to  completely  prove  compliance 
with  the  requirements  without  integrating  the 
two  entities  and  validating  their  combined 
performance  at  the  system  level.  So  it  may  not 
make  sense  to  perform  a  FCA/PCA  on  a 
software  CSCI  prior  to  the  completion  of  all 
the  DT&E  testing.  This  approach  forces  a 
tremendous  load  on  the  government  and  con¬ 
tractor  reviewers  but  may  be  the  only  way  to 
accurately  demonstrate  compliance  with  all 
the  requirements  and  to  document  all  the 
discrepancies  remaining  before  the  govern¬ 
ment  will  accept  the  tested  article. 

10.7  SUMMARY 

Configuration  management  is  essential  to  the 
development  of  mission  critical  computer 
resources.  It  provides  needed  program 
visibility  and  allows  for  the  control  of  the 
software  requirements,  design,  and  final 
product  in  an  orderly  and  logical  manner.  It 
also  provides  and  audit  trail  and  defines  the 
limits  imposed  on  and  by  the  contractor 
during  the  development  of  a  software 


product.  Configuration  management  is  espe¬ 
cially  important  for  software  since  the  physi¬ 
cal  and  functional  characteristics  of  software 
cannot  be  assessed  by  visual  inspection  like 
hardware  [6].  Embracing  the  configuration 
management  discipline  means  making  a  con¬ 
tinuous,  firm  commitment  to  tracking  status 
of  information  that  is  constantly  changing  [7]. 
Without  the  structure  of  a  sound  software 
configuration  management  program,  the 
development  of  complex  software  systems 
would  be  nearly  impossible.  Sound  configura¬ 
tion  management  is  extremely  important  for 
life  cycle  support  of  software. 
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CHAPTER  11 


INDEPENDENT 

VERIFICATION  AND  VALIDATION 


11.1  BACKGROUND 

Independent  Verification  and  Validation 
(IV&V)  of  software  was  born  during  the  early 
days  of  the  space  and  missile  programs.  Both 
NASA  and  the  military  realized  that  software 
developed  for  spacecraft  and  missiles  had  to 
perform  correctly  the  first  time.  For  example, 
if  the  software  did  not  perform  as  required 
during  the  launch  phase  of  a  missile,  there 
were  no  second  chances.  Software  failures 
usually  meant  the  loss  of  the  mission,  the 
launch  vehicle,  and,  if  the  mission  was 
manned,  the  personnel  on  board. 

Because  of  the  importance  and  criticality  of 
spacecraft  software,  money  and  time  were  set 
aside  for  an  independent  organization  or  con¬ 
tractor,  other  than  the  software  developer,  to 
perform  several  functions: 

(a)  Independently  test  the  performance  of  the 
developed  software; 


(b)  Ascertain  that  the  developed  software 
satisfied  all  of  the  system  level  requirements; 

(c)  Independently  develop  a  separate 
software  package  for  those  functions  deemed 
to  be  critical  to  the  success  of  the  mission. 

This  type  of  IV&V  effort  did  not  come  cheap! 
A  full  blown  IV&V  effort  could  sometimes 
exceed  50  percent  of  the  software  develop¬ 
ment  cost  [1].  The  criticality  of  the  missions, 
however,  justified  these  kinds  of  expendi¬ 
tures.  The  same  was  not  true  when  the  IV&V 
philosophy  was  introduced  into  non-missile 
weapon  system  programs.  Many  programs 
failed  to  tailor  the  IV&V  approach  of  the 
space  and  missile  programs  to  the  require¬ 
ments  of  their  own  programs.  Much  of  the 
software  developed  for  aircraft,  tanks,  and 
ships  could  be  classified  as  non-critical.  This 
meant  that  many  software  failures  had  no  im- 
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pact  on  critical  subsystems  and  had  no  effect 
on  crew  safety.  For  example,  a  failure  in  a 
built-in-test  (BIT)  or  other  diagnostic  on-line 
subsystem,  usually  did  not  have  any  bearing 
on  the  success  or  safety  of  a  particular  mission. 
Likewise,  failure  in  a  piece  of  automatic  test 
equipment  (ATE)  or  in  a  data  reduction  pro¬ 
gram  was  not  catastrophic.  Unfortunately, 
large  amounts  of  program  funds  were  need¬ 
lessly  spent  in  the  IV&V  of  non-critical  or 
secondary  software.  This  has  resulted  in  two 
schools  of  thought.  One  school  feels  that 
software  IV&V  is  an  unnecessary  expense  for 
most  weapon  systems  while  the  other  school 
feels  that  IV&V  should  be  performed  on  all 
software  developments.  Neither  is  entirely 
true!  Software  IV&V  is  indeed  an  important 
aspect  of  developing  quality  software,  but  it 
has  to  be  focused  on  those  areas  that  are 
critical  to  the  success  or  safety  of  a  mission. 

Although  IV&V  is  an  effective  management 
tool  in  developing  quality  software,  caution 
must  be  taken  in  placing  too  much  emphasis 
on  it.  IV&V  never  alleviates  the  prime 
contractor’s  responsibility  for  mission  suc¬ 
cess,  system  safety,  or  other  quality  assurance 
practices.  To  be  an  effective  quality  as¬ 


surance  tool,  IV&V  must  complement  and 
reinforce  the  contractor’s  software  engineer¬ 
ing  process,  configuration  management,  and 
qualification  test  functions  [1]. 

Before  discussing  the  scope  and  usefulness  of 
an  IV&V  effort,  it  will  be  useful  to  define 
some  terms. 

11.2  VITRIFICATION 

Verification  is  Computer  Software  Con¬ 
figuration  Item  (CSCI)  oriented.  As  il¬ 
lustrated  in  Figure  11-1,  it  is  the  iterative 
process  of  determining  whether  the  product 
of  .selected  steps  of  the  CSCI  development 
process  fulfills  the  requirements  levied  by 
previous  steps.  Specific  task  areas  that  make 
up  the  CSCI  verification  process  include: 

(a)  Systems  engineering  analytical  activities 
carried  out  to  ensure  that  the  Software  Re¬ 
quirements  Specification  (SRS)  reflects  the 
requirements  allocated  from  the  System 
Specification; 

(b)  Design  evaluation  activities  carried  out  to 
ensure  that  the  CSCI  design  continues  to 
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meet  the  requirements  of  the  SRS  as  the 
design  proceeds  to  greater  levels  of  detail 
(design  verification); 

(c)  Informal  testing  of  the  CSCI  and  its  com¬ 
ponents  carried  out  by  the  developer  to  assist 
in  development,  provide  visibility  of  progress, 
and  prepare  for  formal  testing  (computer  pro¬ 
gram  verification); 

(d)  Formal  testing  of  the  CSCI  (Formal 
Qualification  Test  [FQT])  carried  out  by  the 
developer  in  accordance  with  government  ap¬ 
proved  test  plans  and  procedures  to  verify  that 
the  CSCI  fulfills  the  requirements  of  the  SRS 
and  to  provide  the  basis  for  CSCI  acceptance 
by  the  government  [1]. 

113  VALIDATION 

Validation  is  system  oriented.  It  comprises 
those  evaluation,  integration,  and  test  ac¬ 
tivities  carried  out  at  the  system  level  to  en¬ 
sure  that  the  system  that  is  finally  developed 
satisfies  the  requirements  of  the  System 
Specification.  Specific  validation  tasks  in¬ 
clude: 

(a)  Systems  engineering  activities  carried  out 
to  ensure  that  the  requirements  in  the  system 
specification  accurately  respond  to  the  opera¬ 
tional  needs  called  for  in  the  Statement  of 
Need  (SON); 

(b)  CSCI  integration  activities  carried  out  to 
assemble  and  check  out  previously  qualified 
CSCls  as  a  fully  functioning  system; 

(c)  The  software  aspects  of  .system  validation 
carried  out  during  System  Developmental 
Test  &  Evaluation  (DT&E)  and  Operational 
Test  and  Evaluation  (OT&E)  to  demon.strate 
that  the  completed  system  meets  the  require¬ 
ments  called  for  in  the  .system  specification 
(validating  the  system). 


11.4  CERTIFICATION 

Certification  refers  to  the  using  command’s 
agreement,  at  the  conclusion  of  OT&E,  that 
the  acquired  system  satisfies  its  intended 
operational  mission.  During  OT&E  the  .sys¬ 
tem  undergoes  test  and  evaluation  aimed  at 
assuring  operational  effectiveness  and 
suitability  under  operational  conditions  [2]. 

11.5  THE  I V&V  PROCESS 

AFSC/AFLCP  800-5,  Software  Independent 
Verification  and  Validation  (IV&V),  Draft  [2] 
is  an  excellent  .source  of  information  on  how 
to: 

(a)  Determine  the  need  for  IV&V 

(b)  Establish  the  scope  of  IV&V 

(c)  Define  the  IV&V  tasks 

(d)  Estimate  software  IV&V  cost 

(e)  Select  IV&V  agent. 

Since  this  document  already  addresses  these 
i.ssues,  this  section  will  briefly  summarize  the 
contents  of  AFSC/AFLCP  800-5. 

1 1.5.1  Determining  the  Need  for  IV&V 

Before  establishing  a  software  IV&V  pro¬ 
gram  it  is  necessary  to  determine  whether  the 
system  being  considered  warrants  an  IV&V 
effort.  This  determination  should  be  made 
during  the  system  Demonstration/Validation 
pha.se  by  identifying  and  examining  software 
requirements.  The  identification  and  ex¬ 
amination  of  requirements  should  be  a  by¬ 
product  of  the  System/Segment  Specification 
(SSS)  and  the  preliminary  Software  Require¬ 
ments  Specification  (SRS).  The  require¬ 
ments  within  each  CSCI  should  be  assessed  to 
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determine  if  an  undetected  error  has  the 
potential  for  causing  death  or  personnel  in¬ 
jury,  mission  failure,  or  catastrophic  equip¬ 
ment  loss.  A  determination  should  be  made 
to  see  if  any  of  the  software  development 
effort  is  to  be  considered  medium  to  high  risk 
due  to  technical  reasons  (i.e.,  complexity, 
state-of-the-art,  system  integration,  maturity 
of  tools).  The  CSCls  that  meet  any  of  these 
criteria  should  be  supplemented  with  an 
IV&V  effort.  In  addition,  the  software  safety 
analysis  performed  as  specified  in  MIL-STD- 
882  may  also  determine  the  need  for  IV&V. 

1 1 .5.2  Establishing  the  Scope  of  IV&V 

The  IV&V  effort  should  be  tailored  so  that  it 
is  commensurate  with  the  level  of  criticality  of 
the  software  being  developed.  After  deter¬ 
mining  the  need  for  IV&V,  its  scope  can  be 
established  by  performing  a  criticality  analysis 
of  the  software.  This  analysis  should  be  done 
at  the  CSCI  level  using  the  SRS  and  Interface 
Requirements  Specification  (IRS).  In  many 
cases,  this  level  of  detail  is  not  always  avail¬ 
able  prior  to  the  FSD  contract.  In  these  cases, 
the  analysis  can  be  performed  at  the  SSS  level. 


of  effort  type  contract  until  the  effort  can  be 
better  defined. 

Regardless  of  whether  one  uses  the  SSS  or 
SRS,  the  test  for  criticality  of  softv'are  re¬ 
quirements  is  performed  using  the  following 
six-step  procedure  (See  AFSC/A.FLCP  800-5 
for  detailed  information): 

(a)  List  the  software  requirements; 

(b)  Identify  the  potential  impact  of  un¬ 
detected  software  errors  or  faults  on  the 
software  requirements; 

(c)  Estimate  the  probability  of  occurrence  for 
each  error; 

(d)  Calculate  the  criticality  value  of  the  re¬ 
quirements  as  shown  in  Table  11-1  (See 
ASFC/AFL.CP  800-5  for  details); 

(e)  Calculate  the  overall  criticality  value  for 
the  system  or  for  each  CSCI. 

(f)  Select  the  appropriate  level  of  IV&V 
based  on  system’s/CSCI’s  criticality  value. 


PROBABILITY 

IMPACT  CATEGORIES 

OF 

OCCURENCE 

CATASTROPHIC 

CRITICAL 

MARGINAL 

NEGLIGIBLE 

Frequent 

12 

9 

6 

3 

Probable 

8 

6 

4 

2 

Improbable 

4 

3 

2 

1 

Impossible 

0 

0 

0 

0 

Table  1 1-1  Requirement  Criticality  Values 


Unfortunately,  this  may  lead  to  an  overall 
IV&V  effort  that  is  too  detailed  for  some 
CSCls  or  not  detailed  enough  for  others.  The 
program  manager  can  partially  deal  with  this 
problem  by  placing  the  IV&V  agent  on  a  level 


11.5J  Defining  the  IV&V  Tasks 

Once  the  criticality  values  are  calculated  the 
appropriate  level  of  IV&V  can  be  selected  as 
shown  below: 
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Criticality  Appropriate 

Value  IV&V  Level 


6-12 

3-6 

2-3 

0-2 


I 

II 

III 

None-III 


The  three  levels  of  IV&V  tasks  include 
various  tasks  that  grow  in  scope  and  com¬ 
plexity  as  you  progress  from  one  level  to  the 
next.  The  IV&V  levels  are  defined  as: 


Level  III 


(a)  Evaluation  of  software  documentation; 

(b)  Participation  in  milestone  reviews  and  for¬ 
mal  qualification  testing,  evaluation  of  test 
plans; 

(c)  Identification  of  critical  requirements  and 
design  issues; 

(d)  Monitoring  the  development  process  and 
providing  technical  consultation; 


Level  I 

Same  tasks  as  level  II  plus  the  following: 

(a)  Structural  and  functional  analysis  of  re¬ 
quirements,  design,  and  code; 

(b)  Propose  alternative  designs  for  critical 
design  areas; 

(c)  Redevelop  key  code; 

(d)  Conduct  special  test  in  critical  areas 
beyond  contractor’s  formal  qualification  tests 
(i.e.,  stress  testing,  simulations). 

Each  IV&V  level  is  progressively  more 
detailed  and  comprehensive  than  the  pre¬ 
vious  level.  The  realism  of  the  criticality 
values  depend  on  the  experience  level  and 
background  of  the  people  performing  the 
analysis.  To  select  the  scope  of  IV&V  and  to 
ensure  that  its  results  are  not  biased,  someone 
other  than  the  software  developer  or  the 
potential  IV&V  agent  should  perform  the 
analysis. 


(e)  Evaluation  of  critical  test  results; 

(f)  Performing  selected  audits. 

Level  II 

Same  tasks  as  Level  III  plus  the  following: 

(a)  Analysis  of  selected  critical  functions; 


It  is  important  to  note  that  IV&V  involves 
activities  over  the  entire  software  life  cycle 
and  that  it  is  more  than  just  another  software 
test  activity.  Too  often  the  majority  of  the 
IV&V  effort  doesn’t  start  until  the  software  is 
in  the  test  phase  when  problems  become 
more  visible.  Waiting  until  testing  to  identify 
and  remove  problems  loses  most  of  the 
benefits  associated  with  IV&V. 


(b)  Spot  checking  design  performance; 

(c)  Independently  testing  critical  code; 

(d)  Analysis  of  developer’s  test  results; 

(e)  Independent  evaluation  of  all  software 
problem  reports. 


The  tasks  performed  by  the  IV&V  agent  can 
vary  greatly  in  terms  of  the  costs  versus  the 
benefit  gained  by  the  government.  The  pro¬ 
gram  office  uld  make  provisions  to  ter¬ 
minate  an  IV&V  task  when  its  cost  exceeds  its 
benefit.  The  program  office  should  establish 
criteria  and/or  thresholds  for  terminating 
IV&V  support.  If  the  IV&V  agent  is  the  sup- 


11-5 


Chapter  1 1  Independent  Verification  and  Validation 


porting  command  for  example,  program 
managem^ent  responsibility  transfer  might  be 
chosen  as  the  termination  point.  If  the  IV&  V 
contractor  is  for  level  Ill  tasks,  then  comple¬ 
tion  of  reviews  on  the  operations  and  main¬ 
tenance  manuals  may  be  the  termination 
criteria.  A  termination  clause  for  an  IV&V 
effort  should  be  included  in  all  IV&V  con¬ 
tracts  or  tasking  documents. 

11.5.4  Estimating  Software  IV&V  Costs 

There  are  no  magic  formulas  to  help  predict 
IV&V  costs.  IV&V  cost  estimating  requires 
a  thorough  understanding  of  the  system 
software  and  sound  judgment.  This  section 
provides  some  simple  guidelines  to  follow 
when  estimating  IV&V  costs.  These 
guidelines  are  based  on  historical  trend  data 
and  past  experience. 


development  cost.  From  this  figure  and  the 
software’s  criticality  value,  the  IV&V  cost  as 
a  percentage  of  the  total  software  or  CSCI’s 
development  cost  can  be  estimated.  These 
estimates  may  be  on  the  high  side  because 
they  were  taken  from  programs  that  did  not 
use  DOD-STiJ)-2167A,  Defense  System 
Software  Development  or  MIL-STD-1815A, 
Ada  Programming  Language.  Use  of  these 
two  standards  should  decrease  IV&V  costs. 
Both  standards  increase  the  engineering  dis¬ 
cipline  in  the  software  development  process 
and  this  improves  software  quality  and  makes 
the  IV&V  effort  easier. 

The  above  guidelines  assume  that  the 
software  development  cost  has  been  correctly 
estimated  and  accurately  defined  in  terms  of: 

(a)  Size  (lines  of  code); 


In  general,  the  cost  of  software  IV&V  can 
range  from  10  to  50  percent  of  the  cost  of 
developing  the  software  depending  on  the 
IV&V  level  selected  [1].  The  majority  of 
IV&V  programs  are  usually  at  the  lower  end 
of  this  cost  range.  Figure  11-2  graphically 
depicts  the  relationship  between  the  IV&V 
level  and  cost  as  a  percentage  of  software 
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Fig.  1 1-2  IV&V  Costs  Vs.  Criticality  Values 


(b)  Complexity; 

(c)  Volatility  of  requirements; 

(d)  Programming  language; 

(e)  Quality  of  existing  software  and  documen¬ 
tation  (when  modifying  or  using  existing 
software); 

(g)  Maturity  of  system  software  and  develop¬ 
ment  tools. 

Variations  in  any  of  these  factors  can  impact 
software  development  cost  and  subsequent 
IV&V  cost.  Before  attempting  to  estimate 
IV&V  cost,  it  is  very  important  to  have  a 
thorough  understanding  of  the  software  re¬ 
quirements. 

IV&V  costs  should  seldom,  if  ever,  exceed  50 
percent  of  the  software  development  cost. 
Above  the  50  percent  level,  it  means  that  one 
expects  software  failures  to  have  catastrophic 
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effects  on  the  program  and  that  such  failures 
are  highly  likely  to  occur.  If  this  is  the  case,  it 
probably  means  that  the  application  is  beyond 
the  state-of-the-art  or  at  the  very  least  on  the 
leading  edge  of  technology  and  may  not  be 
ready  to  proceed  into  full  scale  development. 

1 1.5.5  Selecting  IV&V  Agent 

Selecting  an  IV&V  agent  is  the  responsibility 
of  the  program  office.  The  agent  selected 
must  be  autonomous  from  the  software 
developer.  The  IV&V  agent  may  be  a  part  of 
the  prime’s  organization  but  must  report  to  a 
level  above  the  head  of  the  software  develop¬ 
ment  team  (similar  to  any  contractor’s  quality 
assurance  organization).  However,  there  can 
be  significant  benefits  from  using  the  software 
support  activity  (SSA)  as  the  IV&V  agent. 
This  gives  the  SSA  a  vested  interest  in  the 
software  development  since  they  must  sup¬ 
port  the  system  after  system  delivery. 

The  following  guidelines  should  help  in 
selecting  the  best  qualified  IV&V  source: 

(a)  IV&V  Experience  -  IV&V  involves  the  use 
of  specialized  techniques  performed  by  highly 
skilled  personnel  for  detecting  errors  not 
usually  found  by  the  software  develoj^er. 
Therefore,  one  of  the  most  important 
qualifications  of  a  potential  IV&V  agent  is 
experience.  The  IV&V  agent  must  have  a 
strong  background  in  the  latest  IV&V  techni¬ 
ques  and  have  a  successful  track  record.  A 
large  amount  of  IV&V  experience  gives  the 
IV&V  agent  the  intuitive  insight  into  where 
problems  most  likely  occur;  the  ability  to 
quickly  recognize  problem  areas  and  pit  falls; 
expeditiously  recommend  sound  solutions; 
and  experience  in  working  harmoniously  with 
the  soft  are  developer  in  a  pressure  filled 
environi..ent.  It  is  imperative  that  the  agent’s 
IV&V  experience  be  commensurate  with  the 
level  of  IV&V  desired  on  that  application. 


One  does  not  want  an  IV&V  agent  working 
on  a  level  I  IV&V  effort  whose  entire  ex¬ 
perience  is  limited  to  level  III  IV&V. 

(b)  Application  Experience -The  IV&V  agent 
must  have  experience  in  developing  or  per¬ 
forming  IV&V  on  similar  type  systems  and  be 
qualified  in  the  technology  area  under 
development; 

(c)  Personnel  Experience  -  The  management 
and  technical  personnel  must  have  a  strong 
IV&V  background  on  similar  programs  and 
be  proficient  in  using  their  company’s  IV&V 
tools.  In  addition,  they  must  also  have 
detailed  knowledge  of  similar  systems  and  the 
technology  areas  being  developed; 

(d)  IV&V  Tool  Library  -  The  IV&V  agent 
should  have  IV&V  tools  that  are  appropriate 
for  the  specified  IV&V  level.  These  tods 
should  have  been  regularly  used  in  the  past 
and  not  require  inordinately  specialized 
talent  to  use.  The  existence  of  a  large  tool 
library  is  of  little  value  if  the  tools  cannot  be 
easily  used  or  the  IV&V  agent  is  not 
thoroughly  familiar  with  their  use  and  limita¬ 
tions. 

To  determine  the  qualifications  of  an  IV&V 
agent  the  program  manager  has  two  options. 
First,  review  the  agent’s  past  performance  on 
similar  applications.  Second,  perform  a 
software  capability/capacity  review  on  the 
IV&V  agent  as  described  in  Chapter  8. 
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CHAPTER  12 


METRICS 


12.1  INTRODUCTION 

This  chapter  deals  with  the  mechanics  of 
measuring  the  progress  of  a  software  develop¬ 
ment  effort.  The  material  is  addressed  from 
a  top  level  perspective  and  is  typical  of  the 
data  that  is  usually  presented  to  a  program 
manager.  Because  the  data  will  be  summary 
level  data,  additional  data  at  the  next  level  of 
detail  may  be  required  to  support  a  thorough 
review.  The  software  manager  should  have 
more  details  and  ready  access  to  the 
contractor’s  data.  To  receive  metrics  data  re¬ 
quires  that  this  requirement  be  clearly  called 
out  in  the  contract.  It  is  important  to  examine 
the  contractor’s  process  to  determine  what 
data  is  available  and  to  use  the  contractor’s 
system  as  much  as  possible.  The  government 
should  refrain  from  imposing  unnecessary  re¬ 
quirements  on  a  software  developer  such  as 
too  many  metrics,  frequent  reporting  periods, 
or  peculiar  data  formats.  If  the  data  is  not 
available,  it  may  indicate  a  weakness  in  the 
contractor’s  ability  to  properly  manage 
software  development  and  to  provide 
progress  visibility  to  the  program  office. 


12.2  TYPES  OF  METRICS 

There  are  many  different  software  metrics 
available  but  few  of  them  are  proven. 
Software  metrics  are  still  in  various  stages  of 
development  so  their  utility,  timeliness  and 
cost  will  differ  greatly.  Metrics,  however,  can 
be  used  for  timely  and  realistic  adjustments  to 
the  software  development  process  in  order  to 
balance  cost,  schedule  and  performance  ob¬ 
jectives.  In  addition,  metrics  help  assure  the 
delivery  of  a  product  which  will  satisfy  the 
user’s  needs  and  which  may  be  more  easily 
and  economically  supported  and  evolved. 

Software  metrics  may  be  divided  into  three 
areas:  management  metrics,  quality  metrics 
and  process  metrics. 

12.2.1  Management  Metrics 

These  metrics  are  primarily  concerned  with 
indicators  which  help  determine  progress 
against  plan.  The  indicators  are  selected  from 
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drivers  which  have  an  impact  on  the  required 
effort.  Cost  estimating  models  are  used  to 
relate  the  drivers  to  effort,  cost  and  schedule. 
In  addition  to  these  drivers,  a  set  of  product 
progress  indicators  are  selected  to  help  deter¬ 
mine  the  impact  of  changes  in  the  drivers  to 
product  progress  and  as  more  direct  in¬ 
dicators  of  program  progress.  The  trends  in 
these  indicators  support  forecasts  of  future 
progress,  early  trouble  detection  and  realism 
in  plan  adjustments.  Management  metrics 
have  been  applied  effectively  to  programs  at 
several  system  commands  over  the  past  five 
years. 

12.2.2  Quality  Metrics 

These  metrics  are  primarily  concerned  with 
product  attributes  which  affect  performance, 
user  satisfaction,  supportability,  and  ease  of 
change.  Examples  of  quality  metric  indicators 
are  error  density,  reliability,  expandability, 
and  portability.  Although  reliability  and  error 
density  metrics  have  been  in  use  for  some 
time,  more  development  in  the  quality 
metrics’  area  is  required  to  relate  these 
metrics  to  program  success.  A  number  of 
design  and  analysis  tools  which  support  the 
use  of  quality  indicators  are  emerging  in  this 
area.  Two  tools  which  are  useful  to  Ada 
software  development  programs  are  the  Ada 
Test  and  Verification  System  (ATVS), 
developed  by  the  Rome  Air  Development 
Center  and  the  Ada  Measurement  and 
Analysis  Tool  (ADAM  AT),  developed  by  the 
Dynamics  Research  Corp. 

'File  costs  of  applying  quality  metrics  are 
generally  higher  than  those  for  management 
metrics. 

12.2.3  Process  Metrics 

These  metrics  are  concerned  with  those  in¬ 
dicators  that  deal  with  organizations,  tools. 


techniques  and  procedures  and  are  used  to 
develop  and  deliver  the  software  products. 
Good  examples  of  process  indicators  arf*  con¬ 
tained  in  the  Software  Engineering  Institute’s 
(SEI’s)  contractor  capability  assessment 
questionnaire  and  ASD  Pamphlet  800-5, 
Software  Development  Capability  and 
Capacity  Review  [1].  Process  metrics  may  pro¬ 
vide  earlier  indications  of  difficulty  and  the 
need  for  remedial  action.  They  may  provide 
the  only  rich  source  of  feedback  for  process 
improvement.  More  effort,  however,  is  re¬ 
quired  on  the  definition  of  process  indicators 
and  the  relationship  of  the  process  to  program 
success.  A  number  of  activities  at  the  SEI  are 
underway  in  this  area. 

The  costs  of  applying  process  metrics  are  also 
generally  higher  than  those  for  management 
metrics.  The  rest  of  this  chapter  primarily 
addresses  management  metrics. 

12.3  METRICS  APPLICATION 

Uncertainties  and  pressures  for  change  exist 
throughout  all  phases  of  the  acquisition 
process.  Change  in  the  acquisition  process  is 
inevitable  as  development  proceeds  and  re¬ 
quirements  become  better  defined;  as  cost, 
performance  and  schedule  estimates  are 
refined;  and  as  personnel,  machine,  test  and 
support  resource  requirements  are  identified. 

To  support  these  adjustments,  indicators  of 
actual  versus  planned  progress  are  useful. 
However,  the  Full  Scale  Development  (F’SD) 
activities  appear  to  be  the  most  appropriate 
ones  for  application  of  management  metrics. 
The  degree  of  definition,  the  data  availability, 
and  the  criticality  of  decisions  during  these 
activities  is  consistent  with  a  vigorous  meas¬ 
urement  program.  The  Concept  Exploration 
(CE)  and  Demonstration/Validation  (D/V) 
activities  should  provide  good  insights  into 
remaining  risks  and  uncertainties.  7’hese  ac- 
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tivities  also  provide  useful  information  for 
tailoring  FSD  Request  for  Proposal  (RFP) 
requirements  for  metrics  data  and  analysis, 
and  for  source  selection  criteria  and  weight¬ 
ings. 

12J.1  Pre-Solicitation 

The  risks  in  requirements  definition,  design, 
resource  requirements  and  technology  ap¬ 
plication  should  be  defined  during  the  CE  and 
D/V  activities  leading  to  RFP  preparation  and 
source  selection  for  FSD.  The  likely  areas  of 
uncertainty  should  be  identified  sufficiently 
to  provide  some  basis  for  tailoring  the  RFP 
and  source  selection  criteria.  The  tailoring 
should  reflect  the  type  of  data  input  and  the 
processing  and  analysis  required  for  the 
.specific  project.  If  the  early  activities  are  shor¬ 
tened  or  eliminated  as  may  be  the  case  for 
Non-Developmental  Item  (NDI)  .systems  or 
systems  where  few  of  a  kind  are  being 
procured,  tailoring  of  the  data  requirements 
may  be  more  difficult  and  changes  in  data 
needs,  processing  and  analysis  should  be  ex¬ 
pected.  A  larger  set  of  data  inputs  may  initially 
be  required. 

The  measures  called  out  in  the  RFP  v'ill 
generally  be  a  broader  set  than  required  for 
the  project  after  the  source  selection  and 
negotiation  activities  have  been  completed. 
The  information  gathered  during  these  ac¬ 
tivities  on  development  approach,  risk  assess¬ 
ment  i  nd  contractor  process  maturity  will  be 
used  to  tailor  the  data  collection  and  analysis 
to  the  particular  contractor’s  capability  and 
resources. 

The  efforts  prior  to  the  completion  of  the  FSD 
.solicitation  package  should  be  geared  towards 
refining  the  .scope  of  the  project  in  terms  of 
scrubbed  requirements,  realistic  baseline.s,  an 
independent  cost  and  schedule  estimate,  and 
a  risk  reduction  plan.  The  government  should 


establish  an  appropriate  base  and  strategy  for 
contractor  selection  and  generate  a  plan  for 
the  application  of  management  metrics. 

123.2  Government  Model 

The  government  must  have  some  set  of  es¬ 
timating  relationships,  historical  data  or  es¬ 
timating  model  to  develop  an  independent 
cost  and  schedule  estimate.  The  independent 
estimates  are  necessary  to  bring  realism  into 
the  source  selection  process  and  for  the 
negotiation  of  contract  award.  A  database  to 
.support  generation  of  estimates  and  to  update 
cost  estimating  relationships  and  model 
parameters  should  be  maintained  by  the  sys¬ 
tem  commands.  The  collection  of  data  should 
be  done  in  such  a  manner  that  the  u.se  of  data 
across  projects  is  facilitated  and  the  results  are 
meaningful.  The  u.se  of  the  database,  model, 
and  estimating  relationships  carry  over  to  sup¬ 
port  of  the  monitoring  activities  which  take 
place  after  contract  award.  The  historical  ex¬ 
perience  may  be  used  to  set  thresholds  for 
review  and  assess  the  realism  of  plan  adjust¬ 
ments. 

12.3.3  Resource  Needs 

The  cost  of  tracking  software  development 
status  through  use  of  management  metrics  is 
a  function  of  the  size  and  complexity  of  the 
software  development,  the  previous  ex¬ 
perience  of  the  contractor  and  government 
with  the  problem  domain,  and  the  maturity  of 
the  contractor’s  software  development 
process.  Management  metric  data  needs  to  be 
provided  by  the  contractor  and  processed  and 
analyzed  by  both  the  contractor  and  the 
government.  The  contractor  needs  the  data  to 
understand  his  own  progre.ss  and  to  adjust  his 
plans  and  application  of  resources  as  well  as 
to  report  meaningful  progress  to  the  govern¬ 
ment.  Although  the  contractor  should  per¬ 
form  the  bulk  of  the  analysis  and  reporting. 
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experience  has  shown  that  the  government 
must  have  access  to  the  basic  data  and  some 
capability  to  independently  analyze  and 
present  progress  results. 

The  largest  potential  driver  of  government 
effort  is  the  level  of  maturity  of  the  selected 
contractor’s  software  development  process.  A 
contractor  at  low  maturity  levels  will  not  have 
a  weM-dcTined  process,  progress  will  be  more 
difficult  to  determine,  and  generally  meaning¬ 
ful  measures  of  progress  will  not  be  available 
until  later  in  the  development  cycle.  The  ap¬ 
plication  of  metrics  will  more  certainly  be  an 
additional  activity,  and  existing  process  and 
analysis  tools  will  be  rudimentary  or  ad  hoc. 
The  contractor’s  ability  to  provide  analyses  to 
support  government  needs  for  program  status 
information  will  be  limited.  If  metric’s  ap¬ 
plication  is  to  be  useful,  the  requirements  for 
government  efforts  will  increase.  Unless 
there  is  some  process  definition  and  .some 
basis  for  in-process  measurement,  the  use  of 
after-the-fact  product  measures  may  be  the 
only  practical  approach.  Provisions  for  in¬ 
creased  effort  and  schedule  to  support  rework 
likely  will  be  required. 

The  required  government  level  of  effort  to 
monitor  management  metrics  for  program 
with  50-250  thousand  lines  of  source  code  has 
generally  been  between  1  to  3  man  years  per 
year.  As  quality  and  process  metrics’  applica¬ 
tions  grow,  the  level  of  effort  may  increase  by 
1  to  2  man  years  per  year.  However,  as 
software  contractors’  software  development 
processes  mature,  the  type  of  measures  will 
change  and  their  utility  will  increase.  The  re¬ 
quired  government  effort  may  then  decrease 
and  become  more  of  an  audit  function  that 
checks  to  see  that  the  contractor  is  following 
his  defined  process. 

The  government  requires  personnel  skilled  in 
software  engineering  and  development  to 


make  best  use  of  the  metrics’  application.  If  a 
Program  Office  does  not  have  this  capability 
in  house,  support  could  be  provided  through 
Systems  Engineering  and  Technical  Assis¬ 
tance  (SETA)  or  IV&V  contractors.  Ex¬ 
perience  at  the  Naval  Underwater  Systems 
Center  (NUSC)  has  shown  that  a  core  group 
of  metrics  knowledgeable  personnel  who  look 
across  programs  are  a  valuable  asset  to  the 
system  command  and  its  program  offices. 
They  have  the  capability  to  build  or  adapt 
analysis  tools,  maintain  a  data  base  for  histori¬ 
cal  comparisons,  tailor  the  data  needs  to  the 
particular  project,  and  evolve  the  set  of 
metrics  to  better  serve  the  command. 

The  contractor  level  of  effort  for  applying 
management  metrics  has  varied  from  1  to  5 
man  years  per  year.  A  number  of  other  pro¬ 
gram  office/contractor  activities,  such  as  con- 
figuration  management,  technical 
performance  monitoring,  and  cost/perfor¬ 
mance  reporting,  require  data  inputs  process¬ 
ing  and  analysis  that  are  similar  to  the  data 
input,  processing  and  analysis  required  to  sup¬ 
port  the  use  of  management  metrics.  In  addi¬ 
tion  to  the  personnel  effort  required,  analysis 
tools  and  computer  resources  are  also 
needed. 

The  use  of  management  metrics  on  a  program 
should  be  called  for  in  the  solicitation  pack¬ 
age.  The  source  selection  plan  should  include 
criteria  for  evaluating  the  contractor’s 
response. 

12J.4  RFP  and  SOW 

The  RFP  needs  to  describe  a  core  set  of 
measures  which  attempt  to  anticipate  the  per¬ 
tinent  parameters  and  the  degree  of  disag¬ 
gregation  (i.e.,  breakout  into  critical 
components)  of  these  parameters  required  to 
provide  good  insight  into  software  develop¬ 
ment  status  and  support  realistic  program  ad- 
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justment.  The  choice  of  contractor  is  not 
known  at  RFP  time,  and  so  the  core  set  of 
measures  must  be  broad  enough  to  cover  dif¬ 
ferent  development  processes  and  ap¬ 
proaches,  different  levels  of  contractor 
maturity  and  broad  sets  of  possible  risks.  In 
addition,  it  is  difficult  to  predict  ahead  of  time 
how  progress  will  proceed  and  what  questions 
may  be  asked.  Experience  captured  from  pre¬ 
vious  projects  may  be  used  to  adjust  the  core 
set  of  measures  to  anticipate  later  needs. 

123.5  Choice  of  Measures 

Detailed  plans  for  the  software  development 
process  are  formulated  and  submitted  in  the 
FSD  proposals.  Contracts  are  awarded  and 
management  actions  are  guided  by  a  set  of 
plans  for  the  software  development  process 
keyed  to  a  set  of  assumptions.  These  assump¬ 
tions  include  software  size  and  type,  person¬ 
nel  and  machine  resources  and  capabilities;, 
complexity  and  criticality  of  design  and  per¬ 
formance,  and  requirements’  certainty  and 
stability.  These  drivers  and  assumptions  are 
used  to  estimate  cost,  schedule  and  perfor¬ 
mance  and  to  allocate  resources.  The  factors 
which  significantly  affect  cost  and  schedule 
generally  are  those  which  are  identified  as  the 
driving  factors  in  commonly  used  software 
cost  estimating  models.  Keeping  track  of 
these  drivers  provides  good  insight  into  the 
likelihood  of  change  in  planned  progress.  It 
should  also  highlight  the  need  for  tuning  or 
adjusting  plans  to  meet  program  objectives. 

123.6  Use  of  Models  and  Norms 

The  contractor  proposed  plans  and  a.ssump- 
tions  and  the  models  or  historical  data  which 
relate  the  a.ssumptions  to  effort  are  the  base 
against  which  progress  may  be  measured 
during  FSD.  During  the  source  selection, 
reviews  of  each  contractor’s  assumptions  and 
models  will  point  up  risk  areas  and  highlight 


data  which  will  be  valuable  for  verifying  as¬ 
sumptions  and  which  will  be  critical  for  im¬ 
pact  analysis.  The  risk  assessments  and  the 
scoring  of  proposals  provide  keys  to  the  meas¬ 
urement  areas  which  should  receive  most  em¬ 
phasis  and  determine  the  degree  of 
disaggregation  of  the  data  that  is  required.  A 
centralized  activity  for  keeping  track  of  data 
and  cost  estimating  relationships  at  each  sys¬ 
tems  command  is  recommended  to  provide 
historic  data  for  assessing  plan  realism  and 
determining  risks  as  well  as  updating  the  data 
collection  form. 

12.3.7  Process  Maturity 

The  use  of  the  SEI  process  maturity  model  in 
the  Software  Capability  Evaluation  (SCE) 
provides  another  source  of  inputs  for  tailoring 
the  data  set.  The  evaluation  highlights 
strengths  and  weaknesses  of  the  contractor’s 
software  development  process.  This  is  key  to 
selecting  the  metrics  and  level  of  disaggrega¬ 
tion.  In  addition,  the  maturity  level  indicates 
the  degree  to  which  the  measures  may  be 
effective  and  their  likely  timeliness  in  deter¬ 
mining  status  or  aiding  plan  adjustment.  The 
u.se  of  these  evaluation  techniques  also  pro¬ 
vide  a  basis  for  contractor  improvement  of  the 
proposed  process. 

12.3.8  Negotiation 

During  negotiation  the  information  gained 
from  the  source  selection  analysis  and  evalua¬ 
tion  activities  are  used  to  adjust  contract  re¬ 
quirements  for  collection  and  analysis  of 
management  metrics. 

The  key  results  of  the  source  selection 
analysis  and  evaluation  should  be  used  to  ad¬ 
just  the  data  set,  processing  and  analysis 
which  will  be  required  of  the  selected  contrac¬ 
tor.  The  assumptions  which  support  the  level 
of  effort  and  schedule  decisions  should  be 
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recorded.  The  estimating  relationships  or 
models  which  relate  the  assumptions  to  that 
effort  and  schedule  should  also  be  recorded 
along  with  the  reasons  for  differences  be¬ 
tween  assumptions  and  norms.  The 
contractor’s  proposal,  as  adjusted  by  the 
negotiation,  represents  a  plan  for  software 
development.  By  signing  the  contract,  the 
government  chooses  the  contractor’s  plan  for 
the  software  development  effort. 

The  types  of  measures  to  be  employed  and 
their  degree  of  disaggregation  should  be  in¬ 
fluenced  by  the  SCE  report.  The  level  of 
maturity  and  the  strengths  and  weaknesses 
recorded  in  the  report  should  be  used  to  add 
or  adjust  data  requirements.  The  contractor’s 
plan  for  process  improvement  in  response  to 
the  SCE  report  should  be  approved  and 
provisions  for  tracking  progress  should  be  in¬ 
cluded  in  the  contract. 

The  risk  areas  identified  during  source  selec¬ 
tion  should  be  recorded  and  used  to  adjust  the 
data  collection  and  analysis  requirements. 
Contractor  plans  for  risk  reduction  should  be 
approved  and  provisions  for  tracking  progress 
should  be  included  in  the  contract. 

12.3.9  Contract  Monitoring 

The  data  on  drivers  and  assumptions  should 
be  monitored  periodically  to  determine 
progress  against  plan  and  the  need  for  plan 
adjustment.  Care  should  be  taken  to  preserve 
original  baselines  and  assumptions  so  that  ad¬ 
justed  plans  do  not  paint  a  false  picture  of 
progress  history.  In  addition  to  the  data  on 
drivers  and  assumptions,  data  on  product 
progress  is  required  to  provide  more  direct 
indications  of  plan  status  and  progress.  In¬ 
dicators  of  design,  code  and  test  progress  in 
the  form  of  hard  milestones  (rather  than  per¬ 
centage  of  planned  effort  complete)  provide 
"inchstones"  for  determining  program 


progress.  Agreement  on  definitions  of  the 
hard  milestones  should  be  established  before 
contract  award.  Much  of  the  data  should  be 
automatically  generated  by  modern  con¬ 
figuration  management  tools.  Trends  in  the 
data  may  be  used  to  forecast  future  progress 
and  indicate  the  realism  of  proposed  plan 
adjustments.  The  extrapolation  of  the  trend 
data  on  actuals  generally  provides  the  best 
estimate  of  activity  completion. 

The  measures  used  for  keeping  track  of 
progress  are  generally  interdependent.  For 
example  an  increase  in  software  size  should 
be  accompanied  with  an  increase  in  resources 
or  a  change  in  schedule.  These  interdepen¬ 
dencies  may  be  used  to  assess  validity  of 
reporting  and  realism  of  plan  adjustments. 

The  use  of  management  metrics  on  a  number 
of  system  commands  programs  has  proven  to 
be  a  valuable  aid  in  detecting  early  trouble, 
adjusting  plans  realistically  and  forecasting 
future  progress.  In  addition,  metrics  have 
provided  a  basis  for  asking  more  useful  and 
directed  questions  about  the  true  status  and 
well  being  of  the  project. 

123.10  Adjustments  and  Refinements 

Feedback  from  progress  on  the  effort  should 
be  used  to  adjust  and  refine  the  collection  and 
analysis  requirements.  Requirements  may  be 
deleted  in  areas  in  which  progress  is  good  or 
the  level  of  data  aggregation  may  be  in¬ 
creased.  New  requirements  for  collection  and 
analysis  may  be  generated  as  a  result  of  poor 
progress,  special  problems  or  better  under¬ 
standing  of  the  process.  Lessons  learned 
should  be  passed  to  future  projects. 

12.4  PROGRAM  MANAGER’S  METRICS 

The  state  -of-practice  in  software  metrics  is  far 
from  perfect,  but  there  are  a  few  recom- 
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•  Software  Size  and  Cost  Status 

•  Manpower  Application  Status 

•  Cost  and  Schedule  Status 

•  Resource  Margins 

•  Quantitative  Software  Spec  Status 

•  Design  and  Development  Status 

•  Defects/Faults/Errors/Fixes 

•  Software  Problem  Report  Status 

•  Test  Program  Status 

•  Delivery  Status 

Table  12-1  Metrics 

mended  metrics  that  are  appropriate  for  all 
programs.  Metrics  are  generally  relevant  only 
to  an  individual  program  and  indicate  trends 
or  alert  flags;  they  are  not  absolutes.  In  other 
words,  a  specific  value  may  not  describe  ab¬ 
solute  performance  or  cost  but  the  collective 
data  may  provide  a  level  of  confidence  on  the 
range  of  performance  or  cost. 

The  following  material  is  extracted  almost 
entirely  from  the  Air  Force  Systems  Com¬ 
mand,  Electronics  Systems  Division  and 
MITRE  document,  ESD-TR-88-001, 
Software  Management  Metrics  [2] 

Table  12-1  lists  the  metrics  that  should  nor¬ 
mally  be  available,  at  least  monthly,  to  the 
program  manager.  The  basic  method  requires 
an  initial  planned  estimate  as  a  baseline  to 
measure  progress  against.  Actuals  are  then 
tracked  against  the  baseline  plan  and  trends 
or  thresholds  are  analyzed  for  progress  or 
problems. 

Although  there  are  other  metrics  available, 
the  list  in  Table  12-1  is  an  attempt  to  provide 
the  program  manager  with  a  'Top  Ten  List." 

12,4.1  Software  Size  and  Cost  Status 

The  software  size  metric  tracks  the  magnitude 
of  the  software  development  effort  while  the 
cost  metric  tracks  expenditure  of  resources 


(Figure  12-1).  The  metrics  for  each  Com¬ 
puter  Software  Configuration  Item  (CSCI) 
are  normally  tracked  Individually.  The  trend 
should  be  fairly  stable,  although  a  deviation 
may  indicate  a  better  understanding  of  the 
requirements.  One  should  look  for  an  un¬ 
stable  baseline,  requirements  growth,  and 
poor  planning.  The  software  size  metric  is 
generally  the  input  to  a  cost  model  such  as 
COCOMO  [3].  Changes  in  size  require  more 
resources  and  reflect  a  change  in  cost.  The 
cost  model  is  not  shown. 

Rules  of  Thumb  for  Software  Size  and  Cost 
Metrics 

(a)  Month  to  month  estimates  should  not  vary 
by  more  than  5%.  Variation  may  indicate  a 
better  understanding  of  the  requirements  or 
problems  with  the  contractor’s  development 
process. 

(b)  The  costs  of  developing  new  code,  modify¬ 
ing  code,  or  using  existing  code  differ.  Typical 
effective  weights  for  source  lines  of  code 
(SLOC)  are:  New  code  100%,  Modified  code 
50%,  Existing  code  10%.  It  takes  about  twice 
the  effort  to  produce  new  code  as  it  does  to 
modify  existing  code. 

(c)  SLOC  requirements  directly  impact  man- 
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(d)  Tailor  the  metrics  by  tracking; 

-  Equivalent  SLOC  (weighted); 

-  Each  programming  language; 

-  The  metrics  for  each  CSCI  separately; 

-  Object  code  size. 

12.4.2  Manpower  Application  Status 

Tracking  planned  versus  actual  manpower 
loading  provides  visibility  into  future 
schedule  problems  (Figure  12-2).  Resources 
should  be  identified  as  either  experienced 
personnel  and  senior  personnel,  or  as  inex¬ 
perienced  and  junior  personnel.  Tracking  los¬ 
ses  is  also  important,  particularly  if  the  losses 
are  in  key  positions  such  as  the  chief  program¬ 
mer. 

Rules  of  Thumb  for  Manpower  Metrics 

(a)  The  ratio  of  total  to  experienced  personnel 
should  never  exceed  6:1.  A  ratio  of  3:1  is 
typical. 

(b)  Initial  staffing  usually  comprises  about 
25%  of  total  personnel  requirements. 


(c)  The  front  end  should  be  leveraged  with 
more  experienced  personrx.;!. 


(d)  The  development  schedule  depends  on 
the  amount  of  man-months  expended: 

-  Understaffing  is  an  early  sign  of  schedule 
slippage; 

-  If  you  are  behind,  you  can’t  always  catch  up 
by  adding  more  manpower.  Adding  man¬ 
power  may  even  further  delay  the  overall 
schedule; 

-  Judicious  use  of  overtime  may  help; 

-  Additional  use  of  manpower  may  work  but 
only  for  tasks  that  can  be  separated  or  isolated 
with  simple  interfaces; 

(e)  High  turnover  or  loss  of  key  personnel  is 
a  sign  of  problems; 

(f)  Tailor  the  metrics  by  tracking  the  staffing 
for  each: 

-  Development  task; 

-  Skill  (e.g.  Ada,  Data  Base  Management  Sys¬ 
tems,  Artificial  Intelligence); 

-  Organization  (e.g..  Software,  Quality  As¬ 
surance,  Test). 

12.4.3  Cost  and  Schedule  Status 

Financial  reporting  provides  the  status  of 
work  performed  and  actual  cost  of  work  per¬ 
formed  versus  the  plan  (Figure  12-3).  This  is 
the  traditional  Cost/Schedule  and  Control 
System.  Typically,  the  data  reported  to  the 
government  provides  little  visibility  into  the 
software  development  status.  This  usually  oc¬ 
curs  because  of  inadequate  definition  of  the 
software  Work  Breakdown  Structure  (WBS) 
as  well  as  the  level  of  reporting.  The  WBS 
levels  must  provide  reasonable  visibility  to  the 
program  manager.  One  way  to  do  this  is  by 
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defining  a  product  oriented  WBS  for  the 
software. 

Rules  of  Thumb  for  Cost  and  Schedule 
Metrics 

(a)  Be  alert  to  variation  thresholds  exceeding 

10%. 


(b)  Beware  of  efficiencies  that  are  projected 
to  improve.  Past  performance  is  a  measure  of 
productivity  and  efficiency  and  it  is  very  useful 


Fig.  12-3  Software  Cost  and  Schedule 

for  forecasting  estimates  to  complete.  Ef¬ 
ficiencies  don’t  usually  improve  that  much. 


(c)  Tailor  metrics  by  tracking: 

-  Each  CSCI; 

-  Support  software  development. 

12.4.4  Resource  Margins 

Resources  describe  the  hardware  limitations 
of  the  systems.  These  include  Central 
Processing  Unit  (CPU)  throughput,  memory 
size,  and  input/output  channel  capacity  and 
rate  (Figure  12-4).  Resources  can  have  a 
direct  impact  on  software  productivity  and 
design  efficiency.  The  development  resource 
margins  are  as  important  as  the  target  resour¬ 
ces  particularly  for  planning  software  support. 


Rules  of  Thumb  for  Resource  Margin 
Metrics 

(a)  CPU  utilization  should  allow  for  a  50% 
margin  at  delivery  (this  means  that  only  half 
of  the  resource  has  been  used); 

(b)  Memory  utilization  should  allow  50% 
margin  at  delivery; 

(c)  I/O  utilization  (channels  and  data  rates) 
should  allow  50%  margin  at  delivery; 

(d)  For  real-time  systems,  performance  and 
productivity  deteriorates  quickly  above  70% 
utilization; 

(e)  Consider  hardware  resource  limitations 
(e.g.  memory  addressing  as  a  hardware  limit); 

(f)  Resource  utilization  tends  to  increase  with 
time  so  plan  for  expansion; 


(g)  Schedule  and  cost  "bomb"  at  10%  margin 
(in  other  words  you’ll  already  have  seen  an 


Fig.  12-4  Computer  Resource  Margins 


exponential  rise  in  the  effort  required  to 
squeeze,  pare,  re-code,  fix,  etc.); 

(h)  Tailor  metrics  by  tracking: 
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-  According  to  architecture  (e.g.,  multiple 
CPUs); 

-  Average  and  worst  case; 

-  Host  and  Target  equipment; 

12.4.5  Quantitative  Software  Specification 
Status 


(b)  Requirements  growth,  no  matter  how 
small,  will  impact  planned  resources  and 
should  be  contained  from  the  beginning  of  the 
program; 

(c)  Requirements’  uncertainty  leads  to  En¬ 
gineering  Change  Proposals  (ECPs); 


A  baseline  plan  is  initially  estimated  for  the 
quantity  of  discrete  software  requirements  to 
include  both  functional  requirements  and  in- 


Fig.  12-5  Software  Volatility/Action  Items 

terface  requirements.  Actual  progress  is 
tracked  against  the  plan.  This  metric  provides 
visibility  into  the  progress  of  the  requirements 
analysis  as  well  as  the  growth  of  the  require¬ 
ments  in  the  baseline  (Figure  12-5  and  12-6). 

Rules  of  Thumb  for  Specification  Metrics 


(a)  Each  requirement  should  have  a  planned 
completion  date; 


Fig.  12-6  Software  Volatility/Requirements 


(d) Requirements  are  baselined  at  the 
Software  Specification  Review  (SSR); 

(e)  If  the  requirements  are  not  stable  by  the 
Critical  Design  Review  (CDR),  the  program 
is  in  serious  trouble; 

(f)  Requirements  change  after  CDR  will  most 
probably  impact  the  schedule; 

(g)  Phase  incremental  development  to  allow 
the  requirements  to  be  revisited  before  the 
next  increment’s  PDR; 

(h)  Software  action  items  should  not  remain 
open  beyond  60  days. 

12.4.6  Design/Development  Status 

The  contractor  should  describe  in  the 
Software  Development  Plan  the  process  for 
inspections,  walkthroughs  and  internal  design 
reviews.  These  events  can  be  tracked  to 
determine  the  rale  of  progress.  On  a  large 
project  these  events  can  number  in  the 
thousands  (Figures  12-7  and  12-8). 
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Rules  of  Thumb  for  Design  Metrics 

(a)  The  Software  Requirements  Specification 
(SRS)  should  be  complete  before  the 
Software  Specification  Review  (SSR); 


-  Internal  reviews  of  program  design  lan¬ 
guages  (PDLvS); 

(g)  Delaying  development  to  obtain  a  better 
understanding  of  the  requirements  is  usually 
a  wise  decision.  Tradeoffs  may  be  made  to 
reduce  requirements  for  gains  in  schedule; 

(h)  Diverging  from  the  plan  means  that  the 
requirements  are  less  understood.  You  may 
not  be  ready  for  the  SSR; 

(i)  Tailor  the  metrics  by  performing  more 
detailed  tracking,  from  the  SDD  to  the  Com¬ 
puter  Software  Unit  (CSU). 

12.4.7  Defects/Faults/Errors/Fixes 


(b)  The  Software  Design  Document  (SDD) 
should  be  complete  before  the  PDR; 

Rules  of  Thumb  for  Development  Metrics 

(a)  The  CSU  design  should  be  complete 
before  the  CDR; 

(b)  CSC  integration  and  test  must  be  com¬ 
pleted  before  the  Test  Readiness  Review 
(TRR); 

(c)  Diverging  from  plan  may  mean  schedule 
delays; 

(d)  Track  to  cost  model  (e.g.  COCOMO); 

(e)  Estimated  Source  Lines  of  Code  (SLOC) 
produced  per  staff  month  can  be  categorized 
as: 

Easy  Code  250-500 

Moderate  Code  100-250 
Difficult  Code  30-100 

(f)  Tailor  metrics  by  tracking: 

-  Each  CSCI; 


Tracking  the  actual  performance  of  the 
process  provides  some  visibility  into  the 
product  quality.  This  is  only  true  if  the  con¬ 
tractor  has  a  controlled,  repeatable  process. 
Ad  Hoc  software  development  is  not  predict¬ 
able. 

A  defect  is  an  anomaly  in  the  requirements 
and  design.  A  fault  is  an  anomaly  in  im¬ 
plementation  (code).  An  error  is  the  source 
of  a  fault  and  a  fix  is  a  correction  of  an  error. 
Errors  and  fixes  are  tracked  through  the 
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Software  Problem  Reporting  system  (Figure 
12-9) 

Rules  of  Thumb  for  Defects/Faults/Er- 
rors/Fixes/Metrics 


(a)  Defect/fault/error  rates  are  an  early  in¬ 
dication  of  product  reliability. 


(b)  The  contractor’s  process  should  focus  od- 
defect  prevention  and  early  fault  detection. 

12.4.8  Test  Program  Status 

The  entire  test  process  can  be  tracked  from 
planning  through  detection  of  errors  and  cor¬ 
rection  (Figure  12-10).  Quantitative 
measures  can  start  with  a  requirements  and 
test  cross  reference  matrix.  Progress  can  also 
be  measured  on  planned  versus  actual  tests 


Fig.  12-H  Software  Problem  Reports 


12.4.9  Software  Problem  Reports  Status 

Software  Problem  Reports  (SPRs)  provide 
feedback  on  the  correction  of  errors  and  are 
an  indication  of  product  quality  (Fig.  12-11) 

Rules  of  Thumb  for  SPRs 

(a)  Planning  should  account  for  the  iterative 
and  interactive  nature  of  testing; 

(b)  The  number  of  tests  completed  should 
converge  on  the  number  of  tests  planned; 

(c)  Use  trends  to  predict  schedule; 

(d)  Unresolved  problems  should  decrease  to 
zero  as  you  approach  the  TRR  and  again  as 
you  approach  the  Physical  Configuration 
Audit  (PCA); 

(e) The  number  of  SPRs  is  an  indication  of  the 
testing  adequacy  and  the  code  quality.  Too 
many  SPRs  may  indicate  poor  quality;  too  few 
may  mean  inadequate  testing.  It  takes  a 
process  change  to  improve  the  rate  and  the 
quality.  In  today’s  state  of  practice  a  typical 
range  is  5  to  30  SPRs  per  1000  SLOC. 

(f)  If  the  slope  of  open  SPRs  is  positive  then 
problems  are  being  found  faster  than  they  are 
being  fixed.  If  the  slope  is  negative  then  the 
schedule  can  be  predicted. 

(g)  Tailor  the  metrics  by  tracking: 

-  The  number  of  days  SPRs  are  open  (e.g. 
0-30,  30-  60,  60-90,  over  90); 

-  Open  SPRs  by  type  of  software  (e.g.  applica¬ 
tion,  support,  test,  operating  system); 

-  Open  SPRs  by  priority  (e.g.  critical  to  opera¬ 
tion,  critical  for  integration,  other); 
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-  SPR  density  (SPRs  per  1000  SLOC  in 
categories  0-10, 11-20, 21-30,  and  over  30). 

12.4.10  Delivery  Status 

A  good  indicator  of  progress  is  to  track  both 
internal  and  external  incremental  delivery 
status  (Figure  12-12).  For  example,  one  can 
track  internal  delivery  to  the  test  organization 
for  integration  and  test  and  externa!  delivery 
to  an  IV&V  agency.  Often,  early  problems  in 
a  release  may  be  deferred  to  a  later  release. 
This  delay  should  recognize  the  shift  in 
resource  requirements.  Too  many  deferrals 
can  spell  disaster. 

Rules  of  Thumb  for  Delivei^  Status  Metrics 

(a)  The  number  of  CSUs  per  release  should 
remain  stable  (within  10%). 

(b)  Increments  or  "builds"  should 
demonstrate  useful  capabilities  as  early  as 
practical. 

12.5  SUMMARY 

Metrics  should  provide  the  program  manager 
a  tool  for  much  greater  visibility  into  the 
software  development  than  in  the  past.  The 
metrics  discussed  in  this  chapter  focused 
primarily  on  tracking  progress.  There  are 
other  techniques  available  to  monitor  product 
quality,  although  these  are  much  harder  to 


CDR 


Fig.  12-12  Incremental  Release  Count 

quantify.  It  is  important  that  the  contractor 
have  the  right  mechanisms  in  place  for  proper 
discipline  and  commitment  to  quality. 
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13.1  INTRODUCTION 

Improvements  in  software  productivity  are 
coming  slowly.  Present  conventional  techni¬ 
ques  for  software  production  make  use  of 
libraries  of  primitive  functions  or  algorithms 
(See  Figure  13-1).  There  is  little  carryover 
from  past  developments.  The  primary 
development  effort  is  in-line  with  the  delivery 
of  the  software  product  or  capabilities  and  is 
dedicated  to  a  single  system.  There  is  little 
ability  to  leverage  the  development  invest¬ 
ment  and  provide  products  for  use  on  the  next 
system.  The  resulting  system  structure  is 
typified  by  many  unique  modules  with  com¬ 
plex  interfaces  requiring  significant  effort  to 
integrate,  test  and  maintain. 

The  communications  among  the  various 
players  (user,  system  designer,  programmer) 
involved  in  software  development,  acquisi¬ 
tion,  delivery  and  use  is  constrained.  The  con- 
straints  arise  from  a  number  of  factors 
including  geography,  personnel  availability, 
lack  of  software  development  tools,  procure¬ 


ment  and  acquisition  policy  restrictions,  and, 
the  diverse  backgrounds  and  training  of  the 
players.  The  resulting  situation  might  be 
characterized  as  each  player  tossing  require¬ 
ments,  designs,  and  comments  over  a  wall  to 
one  another.  These  constrained  communica¬ 
tions,  and  the  unique  and  customized  im¬ 
plementations  which  result  from  our  present 
ways  of  building  software,  usually  require  that 
the  computer  programmers  play  a  key  role  in 
the  delivery  and  transition  of  the  system  to  the 
user.  In  fact,  in  many  systems  even  the  system 
designers  and  the  system  test  group  have  a 
difficult  time  discovering  critical  failure 
modes  and  correcting  problems  economically 
when  they  are  discovered.  The  need  to  use 
the  software  development  organization 
during  transition  is  often  underestimated; 
especially  when  the  software  development  or¬ 
ganization  is  not  the  prime  contractor.  The 
prime  usually  phases  out  these  programmers 
and  performs  the  final  system  test  and  opera¬ 
tional  test  and  evaluation  without  them.  Un- 
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Fig.  13-1  Conventional  Techniques 


fortunately,  current  software  development 
practice  makes  it  difficult  for  the  prime  to 
perform  these  functions  adequately. 

There  may  be  another  way  to  build  software 
and  overcome  some  of  the  drawbacks  of  our 
present  techniques.  A  model  of  an  alternate 
software  development  technique  is  shown  in 
Figure  13-2.  In  this  model,  showrooms  of 
larger  more  capable  pieces  are  developed  off¬ 
line  for  later  integration  and  use  in  multiple 
systems.  The  in-line  activities  are,  therefore, 
more  heavily  directed  towards  program  in- 
tegr^aion  than  to  design  and  development. 
The  te.sting  e.Tiphasis  is  more  on  system  test¬ 
ing  than  code  or  CSCI  testing.  Such  an  ap¬ 
proach  may  support  the  use  of  com.puter 
aided  design  and  manufacturing 
(CAD/CAM)  for  software.  The  resulting  sys¬ 
tem  structure  is  more  regular,  has  simpler 
interfaces  and  is  easier  to  test.  Less  in-line 


development  is  required  for  the  delivery  of  a 
software  product  or  capabilities.  This  allows 
the  off-line  development  effort  to  support  a 
larger  number  of  systems. 

However,  reusing  software  on  multiple  sys¬ 
tems  is  a  tough  job.  How  can  the  performance 
and  interface  of  the  modules  be  described? 
How  well  can  they  be  tested?  What  is  their  run 
time?  How  are  they  catalogued?  Who  owns 
the  data  rights?  How  are  they  guaranteed? 

The  technology  and  the  management  in¬ 
frastructure  needed  to  fully  exploit  reusability 
is  not  yet  available.  Efforts  in  the  STARS 
program  and  at  the  Software  Engineering  In¬ 
stitute  are  investigating  the  technical  and 
management  advances  required  to  answer 
these  questions  and  to  bring  reusability  closer 
to  practice.  In  the  meantime,  greater  use  of 
best  existing  practice,  tailored  acquisition 
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Fig.  13-2  Reusability 


strategies,  and  improvement  of  tools  and 
techniques  are  required  to  help  manage  the 
software  acquisition  process. 

13.2  SOFTWARE  COST  UNCERTAINTIES 

Because  the  current  nature  of  the  software 
development  process  is  more  like  model  sliop 
or  custom  tailoring  than  like  mass  production, 
software  cost  estimating  accuracy  is  a  veiy 
strong  function  of  the  program  phase.  The 
uncertainty  in  the  software  development 
process  is  captured  in  Figure  13-3  [1].  Errors 
on  the  order  of  4  to  1  are  likely  when  estimat¬ 
ing  software  costs  at  the  start  of  a  project. 

These  early  cost  estimates  require  difficu't 
judgments  of  complexity,  productivity  and 
size.  Experience  indicates  that  the  curv'e 
should  be  one-sided  because  software  costs 
are  never  overestimated;  they  are  always  un¬ 
derestimated. 


How  is  it  then  that  most  cost  estimators  adver¬ 
tise  techniques  which  purport  to  provide  10  to 
20%  accuracy  ?  Most  of  these  estimates  make 
use  of  regression  analyses  and  deal  with  the 
process  from  the  perspective  of  the  right  end 
of  the  cu  rve.  With  present  techniques,  if  you 
want  better  accuracy,  you  have  to  do  some¬ 
thing  to  move  toward  the  greater  experience 
part  of  the  curve. 

This  can  be  done  by  building  prototypes, 
making  use  of  previously  designed  and 
developed  products,  using  commercial  off- 
the-shelf  softvvare,  or  finding  an  acquisition 
strategy  which  includes  a  contract  definition 
phase  or  incremental  development.  A 
strategy  is  needed  to  delay  major  fund  com¬ 
mitments  until  better  software  requirements 
and  implementation  definitions  are  obtained. 

Early  software  cost  estimating  may  be  likened 
to  the  story  of  the  old  farmer  who  had  a  unique 
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Fig.  13-3  Software  Cost  Estimating 


way  of  estimating  the  weight  of  pigs  (See  Fig-  ment  phase,  a  production  phase  and  a  deploy- 
ure  13-4).  "I  don’t  need  any  of  these  new-  ment  phase.  In  most  Command,  Control  and 

fangied  scales  to  do  that,"  he  said.  He  had  an  Communications  (C^)  systems  acquisitions 

easy  way  to  weigh  pigs  without  scales.  He  laid  and  in  many  non-developmental  item  (NDI) 
a  plank  across  a  pail,  put  a  stone  on  one  end,  acquisitions,  the  normal  weapon  life  cycle  is 
a  pig  on  the  other,  and  balanced  the  plank.  compressed  (See  Figure  13-5)  [2].  Although 

Then  he  guessed  the  weight  of  the  stone.  a  concept  exploration  phase  may  take  place. 

From  that  he  could  easily  calculate  the  weight  the  demonstration/validation  phase  is 

of  the  pig-to  three  significant  figures.  With  generally  omitted  and  the  full  scale  develop- 

current  model  shop  techniques,  one  cannot  ment  and  production  phases  are  combined, 

write  better  specifications  or  obtain  better  When  no  demonstration/validation  phase  is 
cost  estimates  without  first  doing  some  part  of  present,  time  somehow  must  be  allowed  for 
the  job  to  be  specified  or  estimated.  definition  and  detailed  design.  Many  acquisi¬ 

tion  schedules  call  for  preliminary  design 
13.3  SOFTWARE  ACQUISITION  CYCLE  reviews  as  soon  as  three  months  after  contract 

award.  The  contractor  is  forced  to  either  ac- 
The  ideal  acquisition  cycle  includes  a  concept  complish  the  top  level  design  in  the  proposal 

exploration  phase,  a  demonstration/valida-  phase  or  in  the  short  time  available  after 

tion  phase,  a  full  scale  engineering  develop-  award. 
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The  use  of  top-down  design  has  been  recom-  ing  better  user  insight  and  lessening  the 

mended  as  an  orderly  and  disciplined  way  to  misunderstanding  that  arises  from  just  look- 

developand  test  software.  For  new  programs,  ing  at  paper  requirements  and  specifications, 

however,  one  cannot  peiform  top  down  They  also  help  reveal  implementation  dif- 

design  and  development  without  some  ficulties  or  constraints, 

knowledge  of  the  bottom  (See  Figure  13-6). 

One  must  have  some  assurance  that  alloca-  Prototypes  generally  make  use  of  special 

tions  of  requirements,  from  one  baseline  to  operating  procedures  and  operating  systems 

the  next,  will  still  be  valid  when  the  lower  level  and  have  limited  interfaces  and  loading.  They 

designs  are  developed.  Historically,  most  Air  have  no  provisions  for  startover,  continuity  of 

Defense  programs  have  run  out  of  processing  operation,  or  maintenance,  and  use  corn- 

resources  as  a  result  of  not  understanding  the  puters  and  code  that  are  not  intended  for  the 

need  to  carefully  sort  tracks  before  attempt-  target  system.  Too  often  the  government  uses 

ing  radar  data  correlation.  Prototypes  of  the  the  information  from  the  prototype  to 

difficult  parts  help  validate  requirement  al-  generate  high  level  system  performance 

locations  and  avoid  breakage  and  rework.  specifications  for  competitive  bid  with  a  new 

set  of  contractors.  Much  of  the  experience 
13.4  PROTOTYPES  gained  from  the  prototype  development  is  lost 

in  the  process  if  the  prototype  developer  is  not 
Prototypes  are  useful  in  two  ways.  They  help  the  winner.  With  today’s  state-of-the-art 

resolve  requirements  uncertainties  by  provid-  software  production  techniques,  it  is  vei^  dif- 


13-5 


Chapter  13  Epilogue 


Idealized  Cycle 

Concept 

Exploration 


Usual  C  Cycle 


Concept 

Exploration 


Demonstration/ 

Validation 


Full-Scale 

Development 


Production 


V. 


Full-Scale  Development/ 
Production 


Deployment 

■  ■ 

Deployment 


Fig.  13-5  System  Acquisition  Cycle 


ficult  to  convert  a  developer’s  experience  into  manage  the  requirements  and  implementa- 
specifications  of  sufficient  quality  to  allow  tion  uncertainties, 
someone  else  to  gain  that  knowledge  and  ex¬ 
perience.  Unless  the  original  players,  both  13.5  SCHEDULES  AND  MANNING 
user  and  development  contractor,  remain  the 

same  and  the  early  prototype  is  very  close  to  The  amount  of  time  the  government  allows 
the  end  item,  a  second  prototype  must  be  for  design,  and  the  rate  at  which  the  contrac- 
built.  Otherwise,  some  strategy  such  as  in-  tor  staffs  the  software  development  activity, 
cremental  development  must  be  used  to  help  have  a  major  impact  on  the  success  of  the 


Fig.  13-6  Top-Down  Design 
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Fig.  13-7  Software  Manpower  Phasing 


program.  Devenney  [3]  examined  major 
acquisition  programs  at  the  Electronic  Sys¬ 
tems  Division  (ESD)  at  Hanscom  AFB,  Mas¬ 
sachusetts.  He  found  that  in  every  case  the 
contractor  manned  up  quickly  and  that  the 
manning  level  was  constant  throughout  the 
program.  The  absence  of  a  demonstra¬ 
tion/validation  phase  and  the  requirement  to 
hold  PDR  in  the  first  few  months  of  the  con¬ 
tract  forced  this  manning  profile.  More  than 
ten  years  ago,  Fioberts  [4]  suggested  a  revised 
manning  profile  and  model  schedule  for  the 
early  design  activities  leading  to  PDR  and 
CDR  (See  Figure  13-7).  The  intent  of  this 
manning  profile  is  to  allow  more  time  for 


design  analysis  and  breadboarding  of  the  dif¬ 
ficult  parts  of  the  job  and  to  perform  some 
work  at  a  level  below  the  level  one  is  attempt¬ 
ing  to  baseline.  This  allows  for  the  validation 
of  the  top  level  design  and  performance  al¬ 
location  before  building  up  the  development 
team.  The  experience  on  several  ESD 
programs  Indicates  that  with  existing  practice 
development  specifications  are  not  com¬ 
pleted  until  10-15  months  after  contract 
award.  This  is  a  true  measure  of  the  time 
needed  for  validating  top  level  designs  (See 
Figure  13-8). 

Over  time  the  schedule  for  the  PDR  has  been 
been  moved  up  and  is  now  generally  six 
months  after  contract  award.  What  do  all 
those  people  do  while  the  top  level  design  is 
being  developed?  They  start  into  detailed 
design  and  coding  in  order  to  meet  schedule. 
This  premature  design  and  code  must  often  be 
redone.  The  changes  and  uncertainties  lead  to 
frustration  and  poor  morale  for  the  team. 

In  updating  his  COCOMO  model  for  Ada 
implications,  Bany  Boehm  has  suggested  a 
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schedule  similar  to  the  one  shown  in  Figure 
13-7.  He  allows  time  before  the  PDR  to  com¬ 
pile  the  PDL  statements,  validate  interfaces, 
and  perform  semantic  checks.  The  longer 
time  produces  a  more  orderly  process  with 
less  breakage  and  rework. 

13.6  TEAM  SIZE  AND  MANAGEMENT 

Large,  complex  software  jobs  often  require 
hundreds  of  people.  An  article  [5]  by  the 
people  who  managed  Sidewinder,  a  very  suc¬ 
cessful  air-to-air  missile,  addresses  the  size 
and  behavior  of  research  and  engineering 
teams.  It  points  out  that  the  nature  of  com¬ 
plex  acquisitions  is  such  that  the  assignment 
of  too  few  people  may  require  an  infinite  time 
to  complete  the  job.  On  the  other  hand,  the 
assignment  of  additional  engineering  person¬ 
nel  above  a  certain  level  may  not  only  propor¬ 
tionately  reduce  total  time,  but  may,  in  fact, 
increase  the  total  time  to  accomplishment 
(See  Figure  13-9). 

The  author  of  the  referenced  article  recalls 
being  asked  in  grade  school  to  solve  the  fol¬ 
lowing  problem:  "If  two  men  can  dig  a  well  in 
eight  hours,  how  long  does  it  take  four  men  to 
dig  the  same  well".  He  "...  can  recall  being 
haunted  by  the  suspicion  that  perhaps  there 
was  only  room  down  the  well  for  two  men;  in 
which  case  the  extra  two  men  might  have 


some  difficulty  in  usefully  contributing".  A 
similar  phenomenon  has  been  reported  by 
Brooks  in  The  Mythical  Man-Month  [6]. 

If  thv  schedule  for  a  previous  job  was  longer 
than  desired  as  the  result  of  over-staffing,  the 
tendency  may  be  to  add  more  people  to  im¬ 
prove  the  schedule  on  the  next  job.  This  will 
happen  if  the  inter-dependent  nature  of  the 
tasks  and  the  over-staffing  are  not  under¬ 
stood.  Care  must  be  taken  in  using  past  ex¬ 
perience  to  set  objectives  for  future  jobs. 

The  size  job  a  given  organization  can  handle 
is  not  only  a  funetion  of  the  available  number 
of  qualified  and  experienced  personnel.  To 
some  extent,  jobs  can  be  balanced  with 
resources.  When  the  job  grows  in  size,  com¬ 
plexity,  and  interdependence,  the  developer 
must  provide  appropriate  infrastructures  (i.e., 
organization  and  documentation)  to  support 
technical  interchange,  progress  reporting, 
and  increased  span  of  control  (See  Figure 
13-10).  If  a  large  job  is  broken  into  pieces  to 
get  it  accomplished,  a  mechanism  must  be 
devised  for  pulling  it  back  together  again. 


Failures  often  occur  when  firms  take  on  larger 
jobs  than  they  have  previously  undertaken. 
They  generally  use  a  small  team  and  face-to- 
face  communications  for  coordination  and 
control.  When  the  number  of  people  grows, 
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there  are  no  established  procedures,  tools  or 
documentation  to  support  the  necessary  tech¬ 
nical  and  management  interchanges. 

13.7  ASSESSING  PERFORMANCE 

There  are  a  number  of  new  techniques  for 
assessing  a  contractor’s  experience,  tools  and 
procedures.  Visits  to  contractor  facilities  by  a 
team  of  experienced  government  software 
personnel  during  source  selections  are  being 
u.sed.  The  team  assesses  and  verifies  contrac¬ 
tor  experience,  maturity  of  procedures, 
quality  management  capabilities  and  job  un¬ 
derstanding.  At  ESD  these  teams  are  called 
"Greybeards."  At  the  Aeronautical  Systems 
Division,  they  are  called  "Capability/Capacity 
Review  Teams"  (See  Chapter  8).  Work  at  the 
Software  Engineering  Institute  and  at 
MITRE  Corporation  [7]  has  led  to  a  process 
for  assessing  the  software  engineering 
capability  of  contractors.  At  ESD  a  short 
software  engineering  exercise  has  been 
designed  to  audit  the  contractor’s  use  of 
proposed  tools  and  procedures. 

In  addition,  a  set  of  metrics  has  been 
developed  for  use  by  the  contractor  during  the 
software  development  process  as  an  aid 
visibility  and  control  [8].  A  number  of  con¬ 
tracts  now  require  the  use  and  reporting  of 
metrics.  The  cost  for  this  should  not  be  high 
since  a  competent  contractor  normally 
generates  metrics  for  his  own  internal  use. 

13.8  MANAGEMENT  GUIDANCE 

Software  development  and  acquisition  is  dif¬ 
ficult  and  will  remain  so  for  several  years. 
Actions  are  required  to  overcome  the 
roadblocks  associated  with  the  present 
process. 

Choose  a  good  contractor.  During  source 
selection  use  in-plant  inspections  (Grey¬ 


beards),  software  exercises,  and  contractor 
capability  assessment  techniques  to  aid  the 
.selection  process.  Include  the  support  re¬ 
quirements  for  these  activities  in  the  RFP. 

Allow  time  for  design  and  iteration.  Don’t 
baseline  without  some  experience  at  lower 
levels  of  design. 

Make  maximum  use  of  off-the-shelf  software. 

Try  to  make  the  job  smaller  and  more 
manageable.  Change  the  requirements  for 
well  defined,  less  critical  functions  to  fit  with 
available  packages. 

Breadboard  and  prototype  the  dilBcult  parts 
of  the  job.  Prototypes  help  the  user  see  what 
he  is  getting  and  help  the  developer  under¬ 
stand  implementation  difficulties. 
Prototypes  generally  are  not  suitable  for  the 
end  product  unless  they  have  been  especially 
designed  for  that  purpose. 

Schedule  preliminary  design  reviews  consis¬ 
tent  with  the  degree  of  validation.  Don’t 
force  contractor  top  level  designs  to  be  ac¬ 
complished  in  the  proposal  stage.  Allow  time 
to  validate  the  top  level  design. 

Apply  discipline  and  new  tools.  Contractors 
should  have  a  good  software  development 
plan,  established  procedures,  and  tools  to  aid 
development,  configuration  management, 
test  generation,  and  status  keeping. 

Require  metrics  for  visibility  and  control. 

The  contractor  slKHild  have  his  own  means  for 
determining  status  and  the  rate  of  progress. 
Trends  are  the  basis  for  predicting  future 
progress.  Impose  the  requirement  to 
generate  and  use  metrics  on  the  contractor. 

Have  more  than  one  phase.  Have  a  place  to 
put  new  requirements  instead  of  impacting 
ongoing  efforts.  Use  the  second  phase  to  deal 
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with  shortfalls  in  the  first  phase.  Often  the 
user  no  longer  wants  the  capabilities  which 
were  thought  desirable  early  in  the  process. 

Deliver  in  useful  increments.  Try  to  keep  the 
deliveries  consistent  with  the  degree  of 
knowledge  one  has  developed  of  require¬ 
ments  and  implementation  difficulties.  To 
avoid  retraining,  design  user  interfaces  so  that 
they  don’t  change  with  each  increment. 

Maintain  schedule;  deliver,  then  add.  Every 
schedule  change  opens  the  door  to  new  re¬ 
quirements.  Delivery  provides  performance 
feedback-necessary  for  further  evolution. 
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ACVC 

Ada  Compiler  Validation  Capability 

Cl 

Configuration  Item 

ADAMAT 

Ada  Measurement  &  Analysis  Tool 

CIDS 

Critical  Item  Development  Spec 

ADP 

Automatic  Data  Processing 

CM 

Configuration  Management 

AFSCP 

Air  Force  System  Command  Pamphlet 

CMP 

Configuration  Management  Plan 

AKLC 

Air  Force  Logistics  Command 

CPU 

Central  Processing  Unit 

AFB 

Air  Force  Base 

CRISD 

Computer  Resources  Integrated 

AFR 

Air  Force  Regulation 

Support  Document 

AIS 

Automated  Information  Systems 

CRWG 

Computer  Resources  Working  Group 

AJPO 

Ada  Joint  Program  Office 

CSC 

Computer  Software  Component 

AMC 

Army  Materiel  Command 

CSCI 

Computer  Software  Configuration  Item 

ANSI 

American  National  Standards  Institute 

CSOM 

Computer  System  Operator’s  Manual 

APSE 

Ada  Programming  Support  Environment 

esu 

Computer  Software  Unit 

ASCII 

American  Standard  Code  for  Information 

CRLCMP 

Computer  Resources  Life  Cycle 

Interchange 

Management  Plan 

ASDP 

Aeronautical  Systems  Division  Pamphlet 

ATE 

Automatic  Test  Equipment 

CRWG 

Computer  Resources  Working  Group 

ATA 

Advanced  Tactical  Aircraft 

ATF 

Advanced  Tactical  Fighter 

DARCOM 

U.S  Army  Material  Development  & 

ATVS 

Ada  Test  and  Verification  System 

Readiness  Command 

AVF 

Ada  Validation  Facility 

DCMC 

Defense  Contract  Management  Command 

AVO 

Ada  Validation  Organization 

DCP 

Decision  Coordination  Paper 

DID 

Data  Item  Description 

BIT 

Built-in-test 

DOD 

Department  of  Defense 

BITE 

Built-in-test  Equipment 

DODD 

Department  of  Defense  Directive 

DOD-STD 

Department  of  Defense  Standard 

CAD/CAM 

Computer  Aided  Design/Computer 

DSB 

Defense  Science  Board 

Aided  Manufacturing 

DT&E 

Developmental  Test  and  Evaluation 

CET 

Capability  Evaluation  Team 

DA' 

Demonstration  and  Validation 

CCB 

Configuration  Control  Board 

ECP 

Engineering  Change  Proposal 

CDR 

Critical  Design  Review 

ECR 

Embedded  Computer  Resources 

CDRL 

Contract  Data  Requirements  List 

EPROM 

Erasable  Programmable  ROM 

CE 

Concept  Exploration 

EEPROM 

Electrically  Eraseable  PROM 

CLIN 

Contract  Line  Item  Number 

ESD 

Electronic  Systems  Division  (USAF) 
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FAR 

Federal  Acquisition  Regulations 

PM 

Program  Manager 

FCA 

Functional  Configuration  Audit 

PMD 

Program  Management  Directive 

FQR 

Formal  Qualification  Review 

PMP 

Program  Management  Plan 

FQT 

Formal  Qualification  Test 

PMRT 

Program  Management 

FSD 

Full  Scale  Development 

Responsibility  Transfer 

FSM 

Firmware  Support  Manual 

FO 

Program  Office 

PROM 

Programmable  Read  Only  Memory 

HOL 

Higher  Qrder  Language 

HWCI 

Hardware  Configuration  Item 

QA 

Quality  Assurance 

1C 

Integrated  Circuit 

RAM 

Random  Access  Memory 

ICBM 

Inter-Continental  Ballistic  Missile 

R&D 

Research  and  Development 

ICS 

Interpretative  Computer  Simulation 

RF 

Radio  Frequency 

ICVVG 

Interface  Control  Working  Croup 

RFF 

Request  for  Proposal 

IDI) 

Interface  Design  Document 

ROM 

Read  Only  Memory 

IEEE 

Institute  of  Electrical  and  Electronics 

Engineering 

SCM 

System  Concejjt  Paper 

ILSP 

Integrated  Logistics  Support  Plan 

SCP 

System  Coneept  Paper 

INS 

Inertial  Navigation  System 

SCRB 

Software  Configuration  Review 

I/O 

Input/Output 

Board 

IOC 

Initial  Operating  Capability 

SDCCR 

Software  Development  Capability 

IR&U 

Independent  Research  and 

Capacity  Review 

Development 

SDF 

Software  Development  Folder  (File) 

IRS 

Interface  Requirement  Specification 

SDL 

Software  Development  Library 

ISA 

Instruction  Set  Architecture 

SDF 

Software  Development  Plan 

IV&V 

Independent  Verification  and 

SDR 

System  Design  Review 

Validation 

SEl 

Software  Engineering  Institute 

SIL 

Systems  Integration  Laboratory 

JLC 

Joint  Logistics  Commanders 

SIF 

Systems  Integration  Facility 

SLOC 

Source  Lines  of  Code 

KAFSE 

Kernal  Ada  Programming  Support 

SON 

Statement  of  Need 

Environment 

SOW 

Statement  of  Work 

SFD 

System  Program  Direetor 

LAN 

Local  Area  Network 

SPM 

Software  Programmer’s  Manual 

SPO 

System  Program  Office 

MCCR 

Mission  Critical  Computer  Resources 

SFR 

Software  Problem  Report 

MIL-STD 

Military  Standard 

SFS 

Software  Product  Specification 

SRR 

vSystems  Requirements  Review 

NCSC 

National  Computer  Security  Center 

SRS 

Software  Requirements  Specification 

NDI 

Non-Developmental  Item 

SSA 

Software  Support  Activity 

NIST 

National  Institute  of  Standards 

SSA 

Source  Selection  Authority 

and  Technology 

SSAC 

Source  Selection  Advisory  Council 

SSEB 

Source  Selection  Evaluation  Board 

OFF 

Operational  Flight  Program 

SSDD 

System/Segment  Design  Document 

O&M 

Operation  and  Maintenance 

sss 

Source  Selection  Plan 

OMK 

Office  of  Management  and  Budget 

SSR 

Software  Specification  Review 

OOl) 

Object  Oriented  Design 

SSS 

vSystem/Segment  Specification 

OS  I) 

Office  of  Secretary  of  Defense 

STARS 

Software  Techm)logy  for  Adaptable  Reliable 

OT&E 

Operational  Test  and  Evaluation 

Systems 

FC 

Personal  Computer 

STD 

Standard 

FCA 

Physical  Configuration  Audit 

S'l’E 

Special  Test  Eciuipmcnt 

FDL 

Program  Design  Language 

STF 

Software  Test  Plan 

FDR 

Preliminary  Design  Review 

STR 

Software  Test  Report 

FDSS 

Post-Deployment  Software  Support 

SUM 

Software  User’s  Manual 

FIDS 

Prime  Item  Development  Spec 

SAV 

Software 
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TADSTAND  Tactical  Digital  Systems  Standard 

T&E  Test  and  Evaluation 

TEMP  Test  and  Evaluation  Master  Plan 

TO  Technical  Order 

TPS  Test  Program  Set 

TRR  Test  Readiness  Review 


UDF 

Unit  Development  Folder 

VDD 

Version  Description  Document 

VUSIC 

Very  High  Speed  Integrated  Circuit 

WBS 

Work  Breakdown  Structure 
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Address 

Specifies  the  location  of  word,  data  or  in¬ 
struction  in  memory. 

Allocated  Baseline 

The  development  specification  which  defines 
performance  requirements  for  each  CSCI. 

Analog 

Being  or  relating  to  a  mechanism  in  which 
data  is  represented  by  continuously  variable 
physical  quantities  [1]. 

Assembler 

A  computer  program  that  translates  assembly 
language  instructions  into  machine  language. 
Typically  one  assembly  language  instruction 
is  translated  into  one  corresponding  machine 
language  instruction.  Both  the  assembly  and 
machine  languages  are  unique  to  a  particular 
computer. 

Assembly  Language 

Assembly  language  allow  the  use  of  ab¬ 
breviated  names  (mnemonics)  for  machine 


language  instructions  and  operands  in  place 
of  binary  (Os  and  Is)  machine  codes. 

Bit 

A  binary  digit  whose  value  is  either  a  1  or  a  0. 

Built-in  Test  Equipment  (BITE) 

Any  device  permanently  mounted  in  the 
prime  equipment  and  used  for  the  express 
purpose  of  testing  the  prime  equipment, 
either  independently  or  in  association  with 
external  test  equipment. 

Byte 

Eight  (8)  bits. 

Buss 

See  Digital  Data  Buss 

Central  Processing  Unit  (CPU) 

Fetches,  decodes  and  executes  the  instruc¬ 
tions  of  the  computer  program.  The  major 
determinant  of  a  compt  ^er’s  execution  speed. 
Sometimes  called  the  "engine". 
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Compilation  or  Compiling 

I'he  translation  process  accomplished  by  a 
compiler. 

Compiler 

A  computer  program  which  translates  a  HOL 
into  machine  language.  The  HOL  statements 
are  called  source  code  and  the  output  of  the 
compiler  is  called  object  code. 

Component 

A  Computet  Software  Component  (CSC)  is  a 
distinct  part  of  a  computer  software  con¬ 
figuration  item  (CSCI).  CSCs  may  be  further 
decomposed  into  other  CSCs  and  Computer 
Software  Units  (CSUs). 

Computer  Program 

A  series  of  instructions  or  statements  in  a 
form  acceptable  to  computer  equipment  and 
designed  to  cause  the  execution  of  an  opera¬ 
tion  or  series  of  operations.  Computer 
programs  include  such  items  as  operating  sys¬ 
tems,  assemblers,  compilers,  interpreters, 
data  management  systems,  utility  programs, 
and  maintenance  or  diagnostic  programs. 
They  also  include  application  progra  ns  such 
as  payroll,  inventory  control,  operational 
flight,  strategic,  tactical,  automatic  test,  crew 
simulator,  and  engineering  analysis  programs. 
Computer  programs  may  be  either  machine- 
dependent  or  machine-independent,  and  may 
be  general  purpose  in  nature  or  designed  to 
satisfy  the  requirements  of  a  specialized 
process  or  particular  users. 

Debugging 

The  processor  locating  and  eliminating  errors 
that  have  been  shown  to  exist  in  a  computer 
program. 

Digital  Data  Buss 

A  group  of  circuits  and  interconnections  be¬ 
tween  two  or  more  devices,  such  as  between 
the  CPU  and  memory  or  between  the  com¬ 


puter  and  external  devices,  that  provide  a 
communication  path  for  digital  data. 

Error  Message 

A  message  printed  out  by  a  computer  after 
detecting  a  programming  error. 

Emulator 

A  combination  of  computer  programs  and 
computer  hardware  that  mimic  the  instruc¬ 
tions  and  execution  speed  of  another  com¬ 
puter  or  system. 

Executive 

The  operating  system  in  an  avionics  suite. 
Expert  Systems 

Systems  that  utilize  artificial  intelligence  (AI) 
to  perform  their  functions. 

Firmware 

Computer  programs  and  data  that  have  been 
written  into  read  only  memories  (ROMs). 

Formal  Qualirication  Review  (FQR) 

A  system  level  configuration  audit  conducted 
after  system  testing  is  completed  to  ensure 
that  the  performance  requirements  of  the  sys¬ 
tem  specification  have  been  met. 

Functional  Baseline 

The  system  requirements.  Provides  basis  for 
contracting  and  controlling  the  system  design. 

Functional  Configuration  Audit  (FCA) 

The  formal  examination  of  test  data  to  deter¬ 
mine  the  functional  characteristics  of  a  CSCI, 
prior  to  acceptance,  to  verify  that  the  item  has 
achieved  the  performance  specified  in  its 
functional  or  allocated  configuration  iden¬ 
tification. 

Higher  Order  Language  (HOL) 

Higher  order  languages  have  been  developed 
in  order  to  make  writing  and  understanding 
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programs  easier.  In  a  HOL,  the  program  is 
written  in  a  series  of  statements  which  typical¬ 
ly  resemble  mathematical  formulas  or 
English  expressions. 

Host  Computer 

The  computer  on  which  a  compiler  executes. 

Initial  Operating  Capability  (IOC) 

The  first  capability  attainment  to  employ  ef¬ 
fectively  a  weapon  system,  an  item  of  equip¬ 
ment,  or  system  of  approved  characteristics, 
and  which  is  manned  or  operated  by  a  trained, 
equipped,  and  supported  military  unit. 

Integrated  Circuit  (IC) 

Tiny  complex  of  electronic  components  and 
their  connections  that  is  produced  in  or  on  a 
small  slice  of  material  such  as  silicon  [l].The 
basic  building  blocks  of  modern  electronics. 

Intermediate  Language 

An  assembly-like  language  used  by  a  compiler 
as  an  interim  step  in  the  process  of  compila¬ 
tion. 

Interpreter 

A  computer  program  that  converts  and  ex¬ 
ecutes  a  HOL  source  program  statement 
directly  into  machine  language,  one  state¬ 
ment  at  a  time. 

Linker 

A  ''omputer  program  that  links  or  ties 
together  programs  that  have  been  separately 
compiled  or  assembled. 

Loader 

The  computer  program  that  loads  the  com¬ 
puter  program  into  memory. 

Machine  Language 

The  binary  codes  (Os  and  Is)  which  are  under¬ 
stood  directly  by  a  computer.  A  typical 
machine  language  instruction  consists  of  an 


operation  code  (or  op-code)  and  one  or  more 
operand  fields.  The  operation  code  specifies 
the  computer  function  (e.g.,  add,  subtract,  test 
for  zero)  to  be  performed  while  the  operand 
fields  specify  where  in  the  computer  the  data 
for  that  function  is  located. 

Maintainability 

The  ability  of  an  item  to  be  reiained  in  or 
restored  to  specified  condition  when  main¬ 
tenance  is  performed  by  personnel  having 
specified  skill  levels,  using  pi  escribed  proce¬ 
dures,  resources,  and  equipment  at  each 
prescribed  level  of  maintenance  and  repair. 

Microprocessor 

A  Central  Processing  Unit  (CPU)  con¬ 
structed  from  one  large  scale  integration 
device  or  chip. 

Model 

A  model  is  a  representation  of  an  actual  or 
conceptual  system  that  involves  mathematics, 
logical  expressions,  or  computer  simulations 
that  can  be  used  to  predict  how  the  system 
might  perform  or  survive  under  various  con¬ 
ditions  of  in  a  range  of  hostile  environments. 

Module 

See  Unit 

Module  Testing 

The  execution  of  a  single  module  to  deter¬ 
mine  its  correctnc.ss  before  the  module  is 
combined  or  integrated  with  other  modules. 

Mnemonic 

Symbolic  names  for  machine  language  in¬ 
structions  which  allow  a  programmer  to 
generate  programs  in  assembly  language 
without  having  to  use  binary  codes. 

Operating  System 

A  computer  program  that  controls  the  execu¬ 
tion  of  other  computer  jnograms  in  a  com- 
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puter.  It  schedules  the  time  when  computer 
programs  are  run,  assigns  memory,  and 
provides  diagnostic  and  accounting  informa¬ 
tion  about  a  program’s  execution. 

Patching 

Making  changes  to  the  machine  code  (object 
code)  representation  of  a  computer  program 
and  by-passing  the  compiler. 

Regression  Testing 

The  testing  of  a  program  to  confirm  that  func¬ 
tions,  that  were  previously  performed  coi  rect- 
ly,  continue  to  perform  correctly  after  a 
change  has  been  made. 

Rehosting 

Modifying  a  computer  program  so  that  it 
operates  on  a  different  host  computer. 

Reliability 

The  probability  that  an  item  will  perform  its 
intended  function  for  a  specified  interval 
under  stated  conditions. 

Retargeting 

Modifying  a  compiler  so  that  it  generates  ob¬ 
ject  code  for  a  different  target  computer. 

Simulation 

A  simulation  is  a  method  for  implementing  a 
model.  It  is  the  process  of  conducting  experi¬ 
ments  with  a  model  to  understanding  system- 
behavior  under  selected  conditions  or  of 
evaluating  various  system  operational 
strategies  within  the  limits  imposed  by 
developmental  or  operational  criteria. 
Simulation  may  include  the  u.se  of  analog  or 
digital  devices,  laboratory  models,  or 
"testbed"  sites. 

Simulator 

A  generic  term  used  to  describe  a  family  of 
equipment  used  to  represent  threat  weapon 
systems  in  development  testing,  operational 


testing  and  training.  A  threat  simulator  has 
one  or  more  characteristics  which,  when 
detected  by  human  senses  or  man-made  sen¬ 
sors,  provide  the  appearance  of  an  actual 
threat  weapon  system  with  a  prescribed  de¬ 
gree  of  fidelity. 

Software 

The  combination  of  computer  programs  or 
instructions  required  to  cause  the  computer 
hardware  to  perform  a  certain  task  or  tasks. 

Stub 

A  stub  takes  ihe  place  of  a  module  that  has  not 
yet  been  coded  or  tested. 

Syntax 

The  rules  for  writing  computer  programs  in  a 
particular  programming  language. 

Syntax  Error 

A  syntax  error  is  generated  when  a  program¬ 
mer  has  violated  the  rules  of  a  particular 
programming  language. 

Target  Computer 

The  computer  for  which  the  compiler 
generates  object  code. 

Testbeds 

A  system  representation  consisting  partially 
of  actual  hardware  and/or  software,  and  par¬ 
tially  of  computer  models  or  prototype 
hardware  and/or  software. 

Test  Program  Set  (TPS) 

Computer  programs  written  in  a  HOT,  usual¬ 
ly  ATLAS,  used  in  conjunction  with  automat¬ 
ic  test  equipment  (ATE)  to  isolate  a  failed 
electronic  subsystem  or  component.  TPSs 
typically  are  used  to  generate  and  inject  test 
patterns  into  digital  circuit  boards  or 
electronic  "black  boxes".  Modern  electronic 
components  are  too  complex  to  be  manually 
tested. 
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Unit 

A  Computer  Software  Unit  (CSU)  is  the 
smallest  testable  element  specified  in  a  Com¬ 
puter  Software  Component  (CSC). 

Validation 

The  process  of  confirming  that  the  software 
(i.e.,  documentation  and  computer  program) 
satisfies  all  user  requirements  when  operating 
in  the  user’s  environment. 

Verification 

The  process  of  confirming  that  the  products 
of  each  software  development  phase  (e.g.,  re¬ 
quirements  analysis,  design,  coding,  testing) 
are  complete,  correct,  and  consistent  with 
respect  to  the  products  of  the  previous  phase. 


Word 

A  data  packet  of  information  for  the  com¬ 
puter.  It  is  usually  composed  of  many  bits, 
The  length  of  a  computer  word  typically  ran¬ 
ges  from  8  bits  for  microprocessors  to  64  or 
more  bits  for  the  larger  computers.  The 
memory  of  a  computer  is  divided  into  seg¬ 
ments  called  words. 

REFERENCES 

l.Webster’s  Ninth  New  Collegiate  Diction¬ 
ary,  Merriam-Webster  Inc,,  Springfield,  MA, 
1984. 
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OUTUNE  OF  PROGRAM  MANAGEMENT  PLAN 


Table  C-1  shows  a  typical  outline  of  a  Program 
Management  Plan  (PMP)  based  on  APR  800- 
2,  Acqumtion  Program  Management,  Attach¬ 
ment  3.  The  other  services  use  a  similar 


1 

Program  Summary  and  Authorization 

* 

2 

Intelligence 

* 

3 

Program  Management 

* 

4 

Systems  Engineering  and  Congiguration 

* 

5 

Test  and  Evaluation 

6 

information  Systems 

7 

Operations 

8 

Civil  Engineering 

9 

Logistics 

* 

10 

Manpower  and  Organization 

11 

Personnel  Training 

12 

Security 

ilr 

13 

Directives,  Specifications.  &  Standards 

* 

Addresses  MCCR 

Table  C-1  PMP  Outline 


outline.  Note  that  six  of  the  sections  must 
address  mission  critical  computer  resources 
(MCCR).  Depending  on  the  program,  there 
may  be  other  sections  that  may  also  need  to 
address  software.  The  discussion  that  follows, 


however,  will  be  limited  to  those  sections 
which  must  address  MCCR. 

C.l  INTELLIGENCE  (Sect  2) 

This  section  includes: 

(a)  Identification  of  the  Threat.  This  para¬ 
graph  should  consist  of  a  listing  of  all  relevant 
threats  that  have  been  obtained  from  the 
Defense  Intelligence  Agency  (DIA)  or  other 
DOD  or  armed  service  agency.  Since 
software  will  probably  be  the  major  player  in 
countering  these  threats,  it  is  important  that 
the  entire  spectrum  of  threats  be  addressed. 

(b)  Identification  of  Relevant  Foreign  Tech¬ 
nology.  Since  the  U.S.  is  no  longer  the  un¬ 
disputed  world  leader  in  technology,  it  is 
important  that  foreign  technology,  especially 
that  of  our  allies,  be  examined  to  determined 
whether  any  of  it  can  be  used  in  the  proposed 
system.  Our  European  allies,  for  example, 
have  embraced  Ada,  the  DOD  standard  com¬ 
puter  language,  with  fervor  and  have  made 
significant  strides  in  the  area  of  Ada  software 
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tools.  Some  of  these  tools,  particularly  com¬ 
pilers  and  Ada  code  analyzers  could  be  useful. 

C.2  PROGRAM  MANAGEMEN  1  (Sect  3) 

This  section  provides  a  description  of  the  ob¬ 
jectives  and  program  strategy  (or  approach) 
in  somewhat  more  detail  than  Section  1 .  Ala 
minimum,  the  schedules  contained  in  this  sec¬ 
tion  should  include  the  following: 

(a)  Operational  system  software  development 
schedules; 

(h)  Training  schedules  including  the  procure¬ 
ment  and  development  of  all  maintenance 
and  operational  crew  trainers; 

(c)  Support  equipment  development  and 
delivery  schedule  including  Al’B  and  other 
software  intensive  special  lest  equipment; 

(d)  Development  and  delivery  schedule  for 
theTPSs  associated  with  the  Al'B. 

C.3  SYSTEMS  ENGINEERING  &  CON¬ 
FIGURATION  MANAGEMENT  (Sect  4) 

This  section  should  describe  when  and  how 
the  functions  of  hardware  and  software  con¬ 
figuration  management  will  be  accomplished 
to  include  the  Configuration  Control  Foard 
(CCB)  and  the  Software  CCF. 


C.4  TEST  AND  EVAEIJATION  (Sect  5) 

This  section  adtlresscs  D'l'&E  and  OT&E 
schedules  including  the  overall  software  test 
and  integration  schedules.  It  also  describes 
the  major  softwtire  support  tools  and  major 
lest  facilities  required. 

C.5  MANPOWER  AND  ORGANIZATION 
(Sect  10) 

'I'liis  section  describes  the  organization  of  the 
program  office  ;md  summarize  the  relation¬ 
ships  and  roles  of  other  military  and  govern¬ 
ment  agencies  and  laboratories.  In  particular, 
it  describes  the  software  organization  and  its 
relation  to  the  other  program  office  organiza¬ 
tions.  It  also  addresses  the  required  software 
manpower  and  skill  levels  along  with  possible 
sources  of  key  software  |)crsonnel. 

C.6  DIRECnVES,  SPECIFICATIONS, 
AND  STANDARDS  (Sect  13) 

'I’his  section  lists  till  the  directives,  specifica¬ 
tions,  and  standtirds  that  will  be  imposed  on 
the  [)rogrtim  including  those  related  to 
software  and  com])ulcrs.  I'or  example,  if  the 
data  buss  standtird,  MIL-S'1'D-1.S53B,  or  the 
Ada  progrtimming  htngutige  standard,  Mll,.- 
STD-KSl.S,  is  inappropritue.  the  reasons  why 
it  is  so  should  be  stated  in  this  section  and  the 
waiver  process  initialed. 
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APPENDIX  D 


TEST  AND  EVALUATION  MASTER  PLAN  OUTLINE 


The  Test  and  Evaluation  Master  Plan 
(TEMP)  is  a  key  program  management  docu¬ 
ment  whose  primary  purpose  is  to  describe  all 
the  necessary  system  Developmental  Test  and 
Evaluation  (DT&E)  and  Operational  Test 
and  Evaluation  (OT&E).  It  relates  program 
schedules,  test  management  structure  and  re¬ 
quired  resources  to  critical  operational  issues, 
critical  technical  characteristics,  required 
operational  characteristics,  evaluation 
criteria  and  decision  milestones.  The  formal 
outline  for  the  TEMP  is  given  in  DOD  Direc¬ 
tive  5()(){).3-M-l  and  it  is  shown  in  Table  D-1. 
The  'FEMP  is  broken  up  into  five  major  sec¬ 
tions  and  three  appendices.  Hardware  and 
software  thresholds  are  to  be  included  in  the 
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TEMP.  It  shall  describe  the  physical  hardware 
tests,  software  tests,  systems  tests,  simula¬ 
tions,  and  any  analyses  needed  to  provide  data 
not  available  through  actual  testing.  In  par¬ 
ticular  the  following  sections  should  specifi¬ 
cally  address  MCCR  issues: 

D,1  SYSTEM  DESCRIPTION  (Part  1.2) 

This  section  briefly  describes  the  system 
design  to  include  the  following  items: 

(a)  Key  features  and  subsystems,  both 
hardware  and  software,  allowing  the  system  to 
perform  its  required  operational  mission; 

(b)  Unique  characteristics  of  the  system  or 
unique  support  concepts  resulting  in  special 
test  and  analyses  requirements  such  as  threat 
simulations  or  other  system  simulators. 

D.2  CRITICAL  TECHNICAL  CHARAC¬ 
TERISTICS  (Part  1.3) 

This  section  lists  in  matrix  format  the  critical 
technical  characteristics  or  the  system  that 
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have  been  evaluated  or  will  be  evaluated 
during  the  remaining  phases  of  development 
testing.  For  MCCR,  this  characteristics  in¬ 
clude:  speed  of  calculation,  memory  utiliza¬ 
tion,  throughput  capability,  reliability 
(growth),  and  response  time. 

D.3  DEVELOPMENTAL  TEST  AND 
EVALUATION  OVERVIEW  (Part  III.l) 

This  section  of  the  TEMP  how  the  planned  (or 
accomplished)  DT&E  will  verify  the  status  of 
the  engineering  design,  verify  that  design  risks 
have  been  minimized,  and  substantiate  the 
achievement  of  technical  performance. 
Specifically  the  narrative  should  identify  the 
degree  to  which  system  hardware  and 
software  design  has  stabilized  so  as  to  reduce 
manufacturing  and  productions  uncertainties. 

D.4  SOFTWARE  TEST  AND  EVALUATION 
(Part  III.4.C) 

In  this  section  all  software  testing  of  MCCR 
required  to  demonstrate  a  quality  product, 
including  post-milestone  III  updates  as  called 
out  in  DOD  Directive  50()().3-M-3,  Software 
Test  and  Evaluation  Manual  and  DOD  Direc¬ 
tive  5000.29,  Management  of  Computer 
Resources  in  Major  Defense  Systems. 

D.5  OT&E  EVENTS,  SCOPE  OF  TESTING, 
AND  SCENARIOS  (Part  V) 

This  section  should  identify  planned  sources 
of  information  (e.g.,  development  testing, 
modeling  and  simulations)  that  may  be  used 
by  the  operational  test  agency  to  supplement 
this  phase  of  OT&E.  Whenever  models  and 
simulations  are  to  be  used,  the  rationale  for 
their  credible  use  should  be  given. 


D.6  TEST  ARTICLES  (Part  V.l.a) 

This  section  should  identify  the  actual  num¬ 
ber  and  timing  requirements  for  all  test  ar¬ 
ticles,  including  key  support  equipment  and 
technical  information  required  for  testing  of 
each  phase.  If  key  subsystems  (components, 
assemblies,  subassemblies  or  software 
modules)  are  to  be  tested  individually,  before 
being  tested  in  the  final  system  configuration, 
identify  each  subsystem  in  the  TEMP  and  the 
quantity  required. 

D.7  THREAT  SYSTEMS/SIMULATORS 
(Part  V.l.d) 

This  section  should  identify  the  type,  number 
and  availability  requirements  for  all  threat 
systems/simulators.  It  should  also  compare 
the  requirements  for  threat  sys¬ 
tems/simulators  with  available  and  projected 
assets  and  their  capabilities.  Major  shortfalls 
should  be  highlighted. 

D.8  SIMULATIONS,  MODELS  AND 
TESTBEDS  (Part  V.l.g) 

This  section  should  identify  the  system 
simulations  required,  including  computer- 
driven  simulation  models  and  hardware-in- 
the-loop  testbeds.  The  rationale  for  their 
credible  use  or  application  must  be  explained 
before  their  use. 

D.9  SPECIAL  REQUIREMENTS  (Part 
V.l.h) 

This  section  should  discuss  requirements  for 
any  significant  non-instrumentation 
capabilities  and  resources  such  as  special  data 
processing  and  special  databases 
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APPENDIX  E 


INTEGRATED  LOGISTICS  SUPPORT  PLAN  (ILSP) 
OUTLINE 


Table  E-1  shows  a  typical  outline  of  an  ILSP. 
Guidance  for  this  document  is  found  in  MIL- 
STD-1369A,  Integrated  Logistics  Support  Pro¬ 
gram  Requirements  (Final  Draft),  The 
following  sections  should  address  Mission 
Critical  Computer  Resources  (MCCR). 

E.l  MAINTENANCE  CONCEPT 

This  section  normally  describes  the  three 
traditional  levels  of  maintenance:  organiza¬ 
tional  or  field  level  maintenance,  inter¬ 
mediate  level  maintenance,  and  depot  level 
maintenance.  Since  modern  weapon  systems 
are  heavily  dependent  on  electronics,  this  sec¬ 
tion  will  discuss  the  types  and  number  of  auto¬ 
matic  test  equipment  (ATE)  contemplated 
for  the  three  levels  of  support;  the  associated 
Test  Program  Sets  (TPS)  which  are  used  in 
conjunction  with  the  ATE  to  isolate  failed 
electronics  components;  and  any  other  com¬ 
puter  and  software-dependent  piece  of 
calibration  or  test  equipment. 


SECTION  I:  GENERAL 

•  System  Description  (QFE  and  associated  SE) 

-  Program  Management  Organization 

and  Responsibilities 

-  Applicable  Documentation 

SECTION  II:  GOALS  AND  STRATEGY 

■  Operation  and  Organization  Concept 

-  Maintenance  Concept 

-  System  Readiness  Objectives 

-  Logistics  Acquisition  Strategy 

-  LSA  Scope  and  Tasks 

-  Supportability  T&E  Concepts/Issues 

-  ILS  Elements 

-  Suppport  Funds 

-  Post  Fielding  Assessment 

SECTION  III:  ILS  MILESTONES  SCHEDULE 

-  ILS  Comprison  to  Program  Milestones 

•  ILS  Elements  (GFE  and  associated  SE) 

-  Assignments,  Responsibilities  and  Events 

Table  E-l  ILSP  Outline 
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E.2  LOGISTICS  ACQUISITION 
STRATEGY 

This  section  summarizes  the  strategy 
described  in  the  Acquisition  Plan.  For 
software  and  computers,  this  section  should 
describe  how  the  logistics  support  software 
will  be  acquired.  For  example,  who  will 
develop  the  ATE  and  TPSs:  the  prime 
developer,  a  subcontractor,  or  another 
government  agency?  Will  the  program  office 
delegate  the  responsibility  for  acquiring  the 
ATE  and  TPSs  to  another  government  agency 
(as  it  is  often  done  in  the  Air  Force)?  Bear  in 
mind  that  development  by  anyone  other  than 
the  prime  contractor  involves  additional  con¬ 
tractual  considerations  (e.g.,  separate  RFP.s, 
source  selections,  and  contracts).  How  will 
calibration  and  special  logistics  support  and 
test  equipment  be  acquired? 

E.3  SUPPORTABILITY  T&E  CON- 
CEPTS/ISSUES 

This  section  describes  the  planned  suppor- 
tability  test  and  evaluation  concept,  its  scope 
and  objectives.  In  particular,  it  addresses  test 
and  evaluation  of  built-in  or  supporting  auto¬ 
matic  operating,  testing,  and  maintenance 
equipment,  and  associated  software. 

E.4  ILS  ELEMENTS 

This  section  addre.sses  all  the  ILS  elements 
tailored  to  each  specific  phase  of  the  acquisi¬ 
tion.  Two  software  intensive  areas  are  training 


and  training  devices  and  computer  resources 
support. 

E.4.1  Training  and  Training  Devices 

This  section  describes  how  training  and  train¬ 
ing  devices  requirements  are  met  and  who  is 
responsible  for  meeting  the.se  requirements. 
Since  this  usually  requires  complex  and  com¬ 
puter-intensive  trainers,  it  should  identify  the 
various  types  and  numbers  of  trainers,  their 
anticipated  location,  and  availability  dates. 
Examples  of  some  of  the  potential  trainers 
required  are;  avionics  maintenance  trainers, 
weapons  loading  trainers,  and  ATE  trainers. 

E.4.2  Computer  Resources  Support 

This  section  de.scribes  the  facilities,  hardware, 
software,  documentation,  manpower,  and 
personnel  needed  to  operate  and  support  the 
embedded  computer  .systems.  Particular  at¬ 
tention  should  be  paid  to  the  ATE  and  the 
associated  TPSs  required  to  isolate  problems 
in  Line  Replaceable  Units  (LRUs)  and 
electronic  circuit  cards.  Organic  maintenance 
and  support  cannot  be  initiated  until  the  ATE 
and  the  TPSs  are  delivered.  This  section 
should  also  describe  the  plans  for  determining 
firmware  and  software  support  and  Post 
Deployment  Software  Support  (PDSS)  pro- 
cedure.s,  requirements,  and  responsibilities.  It 
should  identify  the  requirement  for  prepara¬ 
tion  of  a  Computer  Resources  Life  Cycle 
Management  Plan  (CRLCMP)  as  an  annex  to 
the  ILSP. 
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COMPUTER  RESOURCES  LIFE  CYCLE  MANAGEMENT 
PLAN  (CRLCMP)  OUTLINE 


'I'able  F-1  shows  the  CRLCMP  format  found 
in  Attachment  1 1  of  AI’R  800-14,  Life  Cycle 
Management  of  Computer  Resources  in  Sys¬ 
tems.  Although  the  other  services  do  not  have 
a  standard  format,  (he  formats  used  are  very 
similar. 

V.l  INTRODUCTION 

'Phis  section  slates  the  [nirpose  of  tlie 
CRIX.’MP,  lists  the  approved  system 
nomenclature,  and  lists  the  appropriate  re¬ 
quirements  documents  such  as  the  vStatement 
of  Need  (SON)  and  the  vSystem  Operational 
Concept  (SOC). 

F.2  SYSTEM  CONCIfPTS 

This  section  describes  the  system  operational 
and  sup[)ort  conce[)ts.  It  briefly  describes  the 
mission  of  the  system  with  emphasis  on  com¬ 
puter  resources;  identifies  the  system  func¬ 
tions  which  are  expected  to  require  frequent 
changes  to  accommodate  the  operational  en¬ 
vironment;  and  dc'sci  ioc-  the  hardware  sup¬ 


port  concept  for  the  system  and  for  the 
software. 

F.3  SYSTEM  DESCRIPTION 

Slates  the  purpose  of  the  operational  system 
and  describes  how  the  computer  resources 
relate  to  the  overall  operational  system.  Iden¬ 
tifies  and  rlcscribes  the  characteristics  and 
functions  of  the  processors  in  the  system  and 
the  functions  to  be  implemented  in  software 
or  firmware. 

F.4  COMPUTER  RESOURCES  DESIGN 

'Phis  section  addresses  the  following: 

(a)  System  Architecture  and  Design  -  Iden¬ 
tifies  the  required  hardware  and  software  ar¬ 
chitectures  for  the  system. 

(b)  Product  Improvements  -  Identifies  parts 
of  the  system  which  will  most  likely  require 
future  expansion  (e.g.,  memory  size,  process¬ 
ing  ca[)acity,  number  of  interfaces). 
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Introduction 
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Documentation 

a. 

Overview 

a. 

Types  of  Documents 

b. 

Scope  and  Applicability 

b. 

Data  Rights 

c. 

References 

c. 

Data  Management 

2. 

Systems  Concept 

8, 

Acquisition  Management  Practices 

a. 

Operational  Concept 

a. 

Software  Development  Strategy 

b. 

Support  Concepts 

b. 

Boards  and  Committees 
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Configuration  Management 
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Interoperability 

f. 
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b. 

Contract  Delivery  Schedule 

f. 

Other  Agencies 

c. 

Support  Capabilities 

6. 

Resources 

Appendices 

a. 

Personnel 

A 

Acronyms  and  Abbreviations 

b. 

Facilities 

B 

Glossary  of  Terms 

c. 

Training 

C 

List  of  Key  Personnel 

d. 

Hardware 

D 

CRWG  Charter 
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(c)  Software  Development  Tools  -  Identifies  (d)  Reusability  -  Identifies  and  briefly 

and  describes  the  software  development  tools  describes  any  developed  or  soon  to  be 

and  their  usage  environment.  Indicates  developed  software,  tools,  environments,  or 

whether  they  are  GFE,  GFP,  commercially  facilities  that  may  apply  to  other  current  or 

available  or  contractor  developed.  future  weapon  systems. 
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(e)  Interoperability  -  Briefly  describes  any 
interoperability  requirements  for  the  system 
that  are  implemented  in  computer  resources. 

(f)  Additional  Design  Constraints  -  Briefly 
describes  additional  performance  and  sup¬ 
port  constraints  and  considerations  which 
must  be  translated  into  specified  require¬ 
ments. 

F.5  ORGANIZATION  ROLES 

This  section  describes  the  functional  relation¬ 
ships  among  the  implementing  command,  the 
supporting  command,  the  operating  com¬ 
mand,  and  any  other  participating  command 
or  government  agency. 

F.6  RESOURCES 

This  section  identifies  the  personnel, 
facilities,  training,  hardware,  software,  and  in¬ 
tegrated  logistics  support  requirements  for 
the  system. 

F.7  DOCUMENTATION 

This  section  summarizes  the  documentation 
requirements,  identifies  the  government’s 
software  data  rights,  and  describes  the  plans 
and  procedures  for  managing  the  data. 

F.8  ACQUISITION  MANAGEMENT 
PRACTICES 

This  section  discusses  the  software  develop¬ 
ment  strategy;  identifies  configuration  con¬ 
trol  boards  and  their  interfaces  to  other 
boards  or  committees;  identifies  the  existing 
directives  which  will  govern  configuration 
management  activities;  identifies  the 
schedule  and  participating  organizations  for 
computer  resource  related  reviews  and 
audits;  and  describes  the  schedule  and  par¬ 


ticipating  organizations  for  test  and  evalua¬ 
tion  of  computer  resources. 

F.9  TRANSITION  MANAGEMENT  PRAC- 
TICES 

This  section  discusses  the  procedures  and 
directives  which  will  govern  configuration 
management  and  support  during  and  after  the 
system  is  transferred  from  the  developing  or¬ 
ganization  to  the  using  and  supporting  or¬ 
ganizations. 

F.IO  DEPLOYMENT  MANAGEMENT 
PRACTICES 

This  section  identifies  the  boards  and  com¬ 
mittees  which  are  created  for  the  manage¬ 
ment  of  computer  resources  during  the 
acquisition  phase;  identifies  the  existing 
directives  governing  security  for  the  system; 
and  describes  the  activities  and  major  mile¬ 
stones  associated  with  training  personnel  for 
operating  and  supporting  the  system. 

F.ll  SCHEDULES 

This  section  identifies  the  major  milestones 
associated  with  the  acquisition,  transition, 
and  support  schedules  of  the  system;  iden¬ 
tifies  the  schedules  associated  with  contract 
deliverables;  and  identifies  the  schedule  for 
the  initial  operating  capability  for  the  primary 
computer  resource  support  capabilities. 

F.12  APPENDICES 

The  appendices  will  contain  the  complete 
charter  for  the  Computer  Resouivjs  Working 
Group  (CT<WG)  as  v/ell  as  the  other  items 
indicated  in  the  CRl.CMP  outline  of  Table 
F-1.  It  is  extremely  important  that  the  pro¬ 
gram  manager  carefully  review  the  CRWG 
charter  prior  to  endorsing  it.  The  CRWG 
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must  reflect  the  program  manager’s  objec¬ 
tives  and  philosophy.  Failure  of  the  program 
manager  to  carefully  review  this  charter  can 
lead  to  misunderstandings  and/or  wasted  ef¬ 


forts  later  in  the  program.  In  particular  the 
responsibilities  of  all  the  CRWG  organiza¬ 
tions  must  be  clearly  spelled  out  to  minimize 
turf  battles. 
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SOURCE  SELECTION  PLAN 


Table  G-1  shows  an  outline  of  a  Source  Selec¬ 
tion  Plan  (SSP)  as  given  in  APR  70-15,  Source 
Selection  Policy  and  Procedures.  The  other 
services  use  a  similar  format.  The  following 
sections  that  should  address  mission  critical 
computer  resources. 

G.l  SOURCE  SELECTION  ORGANIZA- 
TION  (Sect  2) 

This  section  describes  the  Source  Selection 
Authority  (SSA),  the  Source  Selection  Ad¬ 
visory  Counsel  (SSAC),  and  the  Source  Selec¬ 
tion  Evaluation  Board  (SSEB  organizations 
and  lists  recommended  key  members  by 
name,  or  by  position  or  functional  area.  It 
must  also  specify  other  government  organiza¬ 
tions  that  will  be  represented  on  the  SSAC 
and  SSEB,  and  include  an  estimate  of  the  total 


1 

Intrcxluction 

2 

Source  Selection  Organization 

3 

Screening  Criteria 

4 

Evaluation  Process 

5 

Evaluation  Criteria 

6 

Acquisition  Strategy 

7 

Schedule  of  Events 

Table  G-1  SSP  Outline 


number  of  personnel  who  will  form  the  mem¬ 
bership,  including  any  advisors. 

It  is  important  that  the  key  software  person  on 
the  SSEB  be  very  knowledgeable  about 
software  acquisition.  Finding  this  individual 
will  not  be  easy  since  knowledgeable,  senior 
software  individuals  are  not  always  readily 
available.  If  this  individual  is  a  member  of  the 
program  office,  then  his  or  her  availability  will 
not  be  a  problem.  However,  if  no  such  in¬ 
dividual  can  be  found  within  the  program  of¬ 
fice,  searching  for  that  individual  must 
become  a  matter  of  high  priority. 

If  the  program  office  is  newly  formed  and 
without  a  knowledgeable  software  individual, 
the  program  manager  must  look  to  either 
other  program  offices,  laboratories,  or  other 
external  organizations  for  an  individual  who 
can  spend  one  to  three  months  (a  typical  dura¬ 
tion  period  for  a  source  selection)  away  from 
his  or  her  current  job.  The  longer  the  source 
selection,  the  more  difficult  it  will  be  to  find 
this  individual.  Ideally  the  senior  software 
person  in  the  source  selection  should  be  the 
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same  individual  who  will  be  managing  the 
effort  once  a  developer  is  selected.  Since  the 
selection  of  the  right  software  deveioper  is 
paramount  to  the  success  of  the  entire  pro¬ 
gram,  finding  a  knowledgeable  software 
evaluator  should  be  the  number  one  priority 
of  the  program  manager. 

G.2  SCREENING  CRITERIA  (Sect  3) 

dliis  section  describes  the  criteria  to  be  used 
to  select  prospective  sources.  The  screening 
criteria  must  be  developed  prior  to  the  official 
publication  of  the  planned  procurement  ef¬ 
fort  in  the  Commerce  Business  Daily  (CBD). 
It  will  be  used  by  procurement  personnel  to 
determine  potential  sources  based  on  the  let¬ 
ters  of  interest  received  in  response  to  the 
CBD  announcement.  Contractors  who  have 
expressed  interest  and  satisfy  the  screening 
criteria  will  receive  copies  of  the  proposal 
package.  Even  if  a  contractor  has  failed  to 
satisfied  the  screening  criteria,  it  may  still 
receive  a  copy  of  the  proposal  if  it  requests  it. 
Ordinarily,  however,  not  too  many  contrac¬ 
tors  wish  to  pursue  a  procurement  in  which 
the  government  doesn’t  feel  it  is  competitive. 

The  screening  criteria  must  include  the  re- 
tjuirement  that  the  sources  solicited  will  have 
(inherently  or  by  subcontracting  or  teaming 
arrangements)  the  management,  financial, 
technical  expertise,  and  security  clearances  to 
design  and  develop  the  system  software. 

G.3  EVALUATION  CRITERIA  (Sect  5) 

This  section  describes  the  specific  evaluation 
criteria  which  will  be  used  to  judge  all  of  the 
(iroposals.  In  generating  the  evaluation 
criteria,  the  program  office  must  ensure  that 
criteria  is  not  overly  restrictive  or  biased 
towards  a  particular  approach  or  technology. 
Remember  that  the  developer  should  have 
the  latitude  to  select  both  the  technology  and 


approach  that  best  satisfies  the  stated  require¬ 
ments.  Unless  there  are  valid  reasons  for 
forcing  the  contractor  to  pursue  a  particular 
path,  such  as  compatibility  with  an  existing 
system,  the  evaluation  criteria  should  reflect 
the  general  requirements  and  not  a  particular 
design. 

G.4  ACQUISITION  STRATEGY  (Sect  6) 

This  section  will  include  a  summary  of  the 
acquisition  strategy,  including  type  of  con- 
tract(s)  proposed,  the  incentives  con¬ 
templated,  milestones  demonstrations 
intended,  and  special  contract  clauses  to  be 
used. 

A  strategy  that  may  be  used  is  to  make  the 
prime  contractor  responsible  for  developing 
all  of  the  system  .loftware  or  subcontracting  it 
out.  Another  approach  may  be  to  direct  the 
prime  contractor  to  subcontract  critical  por¬ 
tions  of  the  software  to  specialized  software 
houses.  For  example,  a  defensive  avionics 
subsystem  may  be  subcontracted  out  to  a  sub¬ 
contractor  specializing  in  electronic  warfare 
systems. 

With  the  increasing  role  played  by  digital 
electronics  subsystems,  an  important  segment 
of  software  is  the  development  of  Test  Pro¬ 
gram  Sets  (TPS)  for  automatic  test  equipment 
(ATE).  Since  TPS  devclooment  is  a  highly 
specialized  business,  a  prime  contractor  nor¬ 
mally  does  no^  have  the  expertise  to  develop 
them.  Even  if  he  did,  it  is  not  prudent  to  put 
so  many  software  eggs  in  one  basket.  It  may 
be  highly  advisable  to  contract  separately  for 
TPS  development.  This  will  mean  a  separate 
source  selection  effort. 

Other  specialized  software  areas  are  trainers 
and  simulators.  Once  again  it  may  be  prudent 
to  contract  these  efforts  separately.  For  a 
major  simulator  such  as  a  weapon  .systems 
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trainer,  the  program  office  may  delegate  this 
responsibility  to  another  organization  or  pro¬ 
gram  office. 

Whatever  strategy  is  selected,  the  most  likely 
approach  will  involve  more  than  one  contrac¬ 
tor.  It  is  important  that  the  proposal  clearly 
state  the  government’s  preference  so  that  the 


contractors  include  in  their  proposals  the  cost 
of  coordinating  with  other  contractors. 

Finally  this  section  should  state  whether  the 
program  office  will  use  an  Independent 
Verification  and  Validation  (IV&V)  or¬ 
ganization  and  their  level  of  involvement  with 
the  software  developers. 
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APPENDIX  H 


SOFimRE  DATA  ITEM  DESCRIPTIONS  (DIDs) 


SPECIFICATION  DIDs 


DI-CMAN-8()008A 

DI-CMAM-80534 

DI-MCCR"80()25A 

DI-MCCR-80()26A 

DI-MCCR-8()029A 

DOCUMENTATION  DIDs 


Systein/Segment  Specification  (SSS) 
vSystem  Segment  Design  Document  (SSDD) 
Software  Requirements  Specification  (SRS) 
Interface  Requirements  Specification  (IRS) 
Software  lYoduct  Specification  (SPS) 


DI-MCCR-8()03()A 

DI-MCCR-80012A 

DI-MCCR-8()()13A 

DI-MCCR-8()014A 

Dl-MCCR-8()015A 

DI-MCCR-80()]7A 

D1-MCCR-8()()U;A 

DI-MCCR-80019A 

ni-MCCR-8()()21A 

DI-MCCR-8()()22A 

D1-MCCR-8()()24A 

DI-MCCR-8()027A 


Software  Development  Plan  (SDP) 

Software  Design  Document  (SDD) 

Version  Description  Document  (VDD) 

Software  Test  Plan  (STP) 

Software  Test  Description  (STD) 

Software  Test  Report  (STR) 

Computer  System’s  Operator’s  Manual  (CSOM) 
Software  User’s  Manual  (SUM) 

Software  Programmer’s  Manual  (SPM) 

Firmware  Support  Manual  (FSM) 

Computer  Resources  Support  Document  (CRISD) 
Interface  Design  Document  (ICD) 
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APPENDIX  I 


APPLICABLE  SOFTWARE  MANAGEMENT  REFERENCES 


FAR  Part  27 
FIRMR  Part  201-7 
DFAR  Part  270 
DOD  DIR  3405.1 
DOD  DIR  3405.2 
DOD  DIR  5000.1 
DOD  DIR  5000.2 
DOD  DIR  5000.29 
DOD  DIR  5000.40 
DOD  DIR  5200.28 
DOD  FAR 
Sup  52-227 
DOD  FAR  Sup  70 
I)OD-SrD-2167A 
DOD-STD-2168 
DOD.STD-5200.28 
DOD  HDBK-287 
DOD  5000 J-M.3 
MIL-STD-152IB 

MIL-STD-1815A 


Federal  Acquisition  Regulation 

Federal  Information  Resource  Management  Regulation 

Defense  Federal  Acquisition  Regulation 

Computer  Programming  Language  Policy 

Use  of  Ada  in  Weapon  Systems 

Major  System  Acquisition 

Defense  Acquisition  Program  Procedures 

Management  of  Computer  Resources  in  Major  Defense  Systems 

Responsibility  for  the  Administration  for  the  DOD  ADP  Program 

Security  Requirements  for  Automated  Information  Systems 

Rights  in  Techni^^al  Data  and  Computer  Software 
Acquisition  of  Computer  Resources 
L'efense  System  Software  Development 
Defense  System  software  Quality  Program 

Department  of  Defense  Trusted  Computer  System  Evaluation  Criteria 
Defense  System  Software  Development  Handbook. 

Software  Test  and  Evaluation  Master  Plan  Outline 
Technical  Reviews  and  Audits  for  Systems,  Equipments, 
and  Computer  Software 
Ada  Programming  Language 
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0MB  CirA-130 
Pub  Law  89-306 
IEEE  P982/D2.4 
IEEE  P982/D5 

APR  800-14 
AFSCP  800-43 
AFSCP  800-14 
AFSCP  800-45 
FSCP/AFLCP  800-5 
ASDP  800-5 
ESD-TR-88-001 


Management  of  Federal  Information  Resources 
Brooks  Bill,  Warner-Nunn  Amendment  and  Implementing  Directives 
"DRAFT'  Standard  for  Measures  to  Produce  Reliable  Software 
"DRAFT'  Guide  for  Use  of  Standard  Measures  to  Produce 
Reliable  Software 

Life  Cycle  Management  of  Computer  Resources  in  Systems 
Software  Management  Indicators 
Software  Quality  Indicators 
"DRAFT'  Software  Risk  Abatement 

"DRAFT'  Software  Independent  Verification  and  Validation  (IV&V) 
Software  Development  Capability/Capacity  Review 
Software  Management  Metrics 
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Index 


INDEX 


Note:  In  most  cases  the  acronym  is  used  in  this 
index.  See  Appendix  A  for  a  list  of  acronyms. 

Ada  3-13,  4-4, 4-5, 6-3, 9-3 
Environment  3-16 
Language  Features  3-15 
Policy  4-4 
Waiver  4-4 
Abstraction  3-15 
AFR  800-14  7-7 
AFSC/AFLCP  800-5  11-3,11-4 
AIS  4-9 
AIS  Policy  4-9 
ANSI  4-6 
Architecture  9-4 
ASCII  3-8 

ASD  Pamphlet  800.5  8-13 
Assembler  3-10 
Assembly  Language  3-10 
Assessing  Performance  13-10 
ATE  6-2 
ATLAS  4-4 
Audits  9-7 
Avionics  3-3 

Baseline 

Allocated  10-6 
Developmental  10-7 
Functional  10-6 
Management  10-6 
Product  10-7 
BASIC  4-5 
Binary  .3-8,  3-10 


Bit  3-8 

Boolean  Logic  3-7 
Bottom-up  Design  13-6 
Brooks  Bill  4-1,  4-9 
Buss  3-9 

CAD/CAM  13-2 

Capability  and  Capacity  Review  8-12 
Capability  Eval  Team  8-15 
CCB  10-7 

CDR  2-9,  5-3, 6-2,  6-7,  6-15, 10-6 

CDRL  8-9, 10-3 

CE  5-2, 6-1,  7-7, 8-1, 8-4 

CET  8-15 

Changeability  9-4 

Cl  Selection  10-2 

CIDS  8-3 

CLlNs  8-8 

CMP  5-5, 8-7, 10-3 

CMS-2  4-4, 4-5 

COBOL  3-11,4-4,4-5 

COCOMO  12-7, 13-7 

Coding  5-9 

Compiler  3-12 

Computer 

Architecture  3-7 
Definition  3-1 
Embedded  3-3,  3-5 
Flight  Control  2-5 
Hardware  3-3 
Instrumentation  2-5 
Mission  Support  2  6 
Navigation  2-5 
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Index 


Computer  (Continued) 

Resources  3-4 
ISA  3-8, 3-9 
Configuration 
Audit  10-11 
Control  10-3 
Control  Process  10-9 
Functional  10-2 
Physical  10-2 
Status  Accounting  10-10 
Configuration  Control  Board  10-3 
Configuration  Management  3-6, 6-13, 7-10, 10-1 
Configuration  Review  Board  10-4 
Contracts  8-6 
Air  Force  8-6 
Evaluating  Proposals  8-12 
Evaluation  8-11 
Fees  9-4 
Incentives  9-4 
Instruction  to  Offerors  8-7 
Proposal  Evaluation  8-8 
Control  Systems  9-5 
Copyright  4-8 
Cost 

Estimating  8-7, 11-6 
Hardware  2-3, 2-4 
IV&V  11-1, 11-7 
Life  Cycle  9-4 

Post  Deployment  2-3, 7-12, 7-7 
Problems  9-6 
Software  2-3, 7-2, 7-3, 13-3 
Software  Fixes  6-5 
Status  12-7 
COTS  8-9 
CPU  3-2 
CRISD  5-11 

CRLCMP  5-5, 7-7, 7-8, 8-3, 8-4, 8-6, 8-7 
CRWG  8-4, 10-5 
CSC  5-5, 6-3, 10-3 
Integration  12-8 

CSCI  5-5,  5-10, 6-4, 10-2, 10-3, 11-2 
CSOM  5-11 
CSU  5-3,  5-5, 10-3 

Data  Analysis  Tools  6-14 
Data  Reduction  Tools  6-14 
Data  Rights  4-8,  4-9 
Defense  Science  Board  4-7 
Demming  Philosophy  9-5 
Design 

Detailed  5-8 


Design  (Continued) 

Metrics  12-10 
Preliminary  5-7 
Development  Model  13-4 
DIDs  8-7,  Appendbc  H 

Documentation  2-9,  3-6, 3-7, 5-8, 5-9,  7-10, 9-2, 9-8 
DOD  Directive 

3405.1  4-4, 4-5 

3405.2  3-17, 4-4 
5000.29  4-2, 4-7 
5000.3-M-3  6-2 

DOD  Instruction 
5000.31  4-4 
7920  4-9 
DOD-STD 

2167A  4-4,  5-1,  5-5,  5-8,  5-11,  8-7, 9-4, 11-6 
2168  4-4, 5-14 
480A  10-1 
5200.28  8-6 

DT&E  6-2, 6-4, 6-10, 11-3 
DA'  5-2,8-1,10-2 

ECP  10-4, 12-10 
EEPROM  3-6 
Emulation  6-4 

Engineering  Change  Proposals  10-4, 12-5 
EPROM  3-6 

Evolutionary  Development  9-4,9-10 
Exception  Handling  3-15 

FAR  3-5, 4-9 
FCA  5-4, 10-11 
Feasibility  Studies  8-4 
Firmware  3-5, 8-7 
FORTRAN  3-11,4-4,4-5 
FQT  11-2 
FOR  5-3,  5-12 
FSD  5-2, 8-1, 8-7, 10-5 
FSM  5-11 

Functional  Analysis  5-2 

Generics  3-16 
GFE  9-6 
GFI  9-6 
Gist  9-9 

HOL  3-11,4-3,8-4 
Human  Factors  3-14 

I/O  3-2 
IC  3-6,  4-6 
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Index 


ICWG  10>5 
IDD  5-7 

ILSP  7-8, 8-2,  Appendix  E 
Incremental  Development  9-6, 9-10 
Information  Hiding  3-15 
Inspections  9-7 

Instruction  Set  Ar-  hitecture  3-8, 3-9, 4-7 
Integrated  Circuits  2-2, 3-2, 4-6 
Integration  9-8 
CSC  12-4 
Interface 

Controls  9-3, 10-5 
Definition  8-3 
Interfaces  8-12 
IRS  5-4, 6-2, 8-3, 8-4, 10-6 
ISA  3-8,  3-9,  4-7 
ISO  4-6 

IV&V  6-7,6-15,8-5,8-8, 11-1 
Levels  11-5 
Need  11-3 
Scope  11-4 
Selecting  Agent  11-7 
Tasks  11-5 

JOVIAL  4-4,  4-5 

Language  3-9, 8-8 
See  also: 

Ada 

Atlas 

Basic 

C 

CMS-2 

COBOL 

FORTRAN 

JOVIAL 

PASCAL 

SPC/1 

TACPOL 

License  Agreements  4-9 
Localization  3-15 

Machine  Language  3-10 
Management 

Baseline  10-6 
Checklist  9-11 
Guidance  13-9 
Guidelines  9-3, 9-5 
IV&V  11-1 
Metrics  12-1 
Monitor  8-4 


PDSS  Guidance  7-11 
Planning  8-1 
Manning  13-6, 13-7 
Margins  9-6 
MCCR  4-1, 4-2 

Management  Steering  Committe  4-2 
Policy  4-3 

Memory  3-3, 8-12, 9-6 
Metrics  8-7, 9-6, 12-1 

Adjustments  and  Refinements  12-6 
Application  12-2 
Choice  12-5 

Contract  Monitoring  12-6 
Cost  12-7 

Delivery  Status  12-13 
Development  12-10 
Government  Model  12-3 
Management  12-1 
Manpower  12-8 
Negotiation  12-5 
Pre-Solicitation  12-3 
Process  12-2 
Process  Maturity  12-5 
Program  Manager’s  12-6 
Resource  Needs  12-3 
Resources  12-9 
RFP/SOW  12-4 
Size  12-7 
Types  12-1 
Quality  12-2 

Use  of  Models/Norms  12-6 
Microprocessor  2-2 
MIL-STD 

1521  b  5-3, 9-7 
1750A  3-9,  4-3,  4-7 
1815A  4-6, 11-7 
483A  10-1, 10-2 
882  11-4 
MITRE  8-14 
Models  6-1 
Modularity  3-15,9-4 

NCSC  8-6 
Noise  6-12 

Object  Oriented  Design  6-2,  9-3 
Object  Oriented  Programming  9-3 
OFP  8-8 

OOD  2-7, 5-7,  9-3 
Operating  Systems  8-12 
Operational  Concept  Analysis  5-4 
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Index 


OPNAVINST  5200.28  7-7 

Requirements  (Continued) 

Orange  Book  8-6 

Analysis/Design  5-3 

OT&E  6-2,  6-4, 6-13, 11-3 

Definition  8-3, 9-4, 9-8 

Packages  3-15 

Refinement  5-4 

PASCAL  4-5 

Specification  12-10 

Patches  6-16 

Testability  9-7 

Patent  4-9 

Traceability  5-1, 9-2, 9- 

PCA  5-3, 5-11, 10-11, 12-12 

Resource  Planning  9-3 

PUL  3-17,6-2,12-11, 13-8 

Resources  Leverage  9-3 

PDR  2-9, 5-3, 5-6, 6-2, 6-7, 6-15, 10-6, 12-10, 13-7 

Restricted  Rights  4-9, 8-9 

PDSS  4-7,7-15,8-4 

Reuse  8-7, 13-3 

Air  Force  7-8 

Reviews  5-2, 9-3, 9-6,  9-7 

Army  7-8 

RFP  8-7, 10-3, 12-4 

Concept  8-4 

Draft  8-9 

Corrections  7-11 

Risk  5-4,  8-4, 8-7, 9-6, 9-7 

Distribution  of  Corrections  7-11 

ROM  3-6 

Documentation  7-10,7-11 

RSL/EVS  9-8 

Enhancements  7-10 

Evaluation  of  Complaints  7-10 

SADT  9-9 

Funding  7-9, 7-3 

SCE  8-14, 12-5, 12-6 

Integration  Testing  7-10 

Schedule  9-8 

Interoperability  Testing  7-10 

Problems  9-6 

IV&V  7-9 

Schedules  9-3, 13-8 

JLC  Workshop  7-2 

Scheduling  6-3 

Location  7-8 

SCM  10-1 

Management  Concerns  7-4 

Audit  10-11 

Management  Guidance  7-11 

Class  1  10-4,10-5 

Management  perceptions  7-3 

Class  11  10-4,10-5 

Marine  Corps  7-8 

Control  10-1 

Navy  7-8 

Identification  10-1 

Organization  Chain  7-8 

Library  10-10 

Policy  7-9 

Status  Accounting  10- 

Process  7-7 

SCRB  10-7 

Software  Environment  7-9 

SDCCR  8-12 

Strategy  7-8 

Factors  8-13 

SW  Engineering  Change  7-10 

Team  8-13 

What  is  7-5 

SDD  5-7 

Peopleware  3-6 

SDF  5-8 

PIDS  8-3 

SDP  6-2,  8-6, 8-7, 10-3 

PMP  8-1 

SDR  5-2,  8-3, 10-2, 10-6 

Problem  Reporting  10-8 

Security  8-6 

Product  Baseline  5-11 

Accreditation  8-6 

PROM  3-6 

Certification  8-6 

Productivity  3-10, 3-13 

SEl  8-14,9-5,4-7 

Prototype  5-6,  8-8,  9-6, 9-8, 13-5 

SETA  12-4 

PSL/PSA  9-8 

SIF  6-9 

Real-time  2-8,  3-2, 9-4 

SIL  6-4, 6-12 

Simulation  6-11 

Requirements 

Simulators  6-1 

Analysis  5-6 

Environment  6-11 

Analysis  Tools  6-14 

System  6-11 
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Index 


Sizing  8-7 
SLOC  12-7, 12-8 
Software 

Acquisition  Cycle  13-4 
Productivity  13-1 

Software  Capability  Evaluation  8-14 

Software  Development  5-1, 5-4 

Software  Development  Folder  10-11 

Software  Development  Library  10-10 

Software  Engineering  2-1,  2-7,  3-13,  3-15, 4-6,  5-1 

Software  Engineering  Institute  4-7 

Software  Errors  6-5 

SON  11-3 

Source  Program  5-9 

Source  Selection  8-9 

Source  Selection  Org  anization  8-10 

Special  Requirements  8-9 

SSA  8-6 

SSAC  8-7 

SSP  8-6 

SOW  5-13,8-2,8-8 
Special  Interest  Items  9-6 
Specifications 
C-5  5-11 
System  5-3 
Type  A  5-4 
SPL/1  4-5 
SPM  5-11 
SPR  12-12 
SPS  5-11 

SQPP  Appendix  H 
SRR  5-2, 10-6 

SRS  5-5,  6-2, 8-3, 8-7, 10-6, 11-3,  11-4, 12-11 

SSA  8-6 

SSAC  8-11 

SSDD  8-3 

SSEB  8-11 

SSR  5-3,5-5,12-10 

SSS  5-4,  8-3,  8-4,  11-4 

Standards  2-7,  2-9,  3-9,  4-3,  4-8,  5-1,  5-2,  5-3, 5-5, 
6-2,  6-4,  6-12, 8-8, 8-14 
STARS  4-7,  13-2 
Statement  of  Need  1 1-3 
Static  Analysis  Tools  6-11 
STP  5-7,6-2,6-13 
STR  5-11 

Structured  Design  9-2 
Subcontracts  8-7 
SUM  5-11 
Support 

Organic  8-12 


Support  (Continued) 

Resources  6-1 
Studies  8-4 
Vendor  8-12 
Facilities  3-7 

System/Segment  Designs  5-3 

TACPOL  4-4 
Tailoring  5-12 
9-2  Tasking  3-16 
Teaming  8-7 
TEMP  5-3,  6-1,  8-2,  8-5 
Test  5-9 

Against  Requirements  9-7 

Black  Box  6-8 

Bottom-Up  6-10 

Debugging  6-14 

Desk  Checking  6-7 

Facilities  6-12 

Formal  5-10 

Hot  Bench  6-4, 6-12 

Human  6-6 

Informal  5-10, 6-3 

Inspections  6-6, 9-2 

Integration  5-10,5-12,6-4,6-12 

PDSS  7-10 

Peer  Ratings  6-7 

Planning  6-1 

Resources  6-1,  6-12 

Software  Only  6-8 

Software  System  6-8 

Support  6-1 

Tools  6-13 

Top-Down  6-10 

Types  6-6 

Walk-Throughs  6-7,  9-2 
While  Box  6-9 
Testing 

Formal  11-3 
Informal  11-3 
Text  Editors  6-14 
Thix)ughpul  3-9,8-12,9-6 
Tools  8-5 

Debugging  6-14 
Development  8-12,9-2,9-6 
Specification  Development  9-8 
Top-Down  Design  13-5 
TPS  8-8 

Trade  Secret  4-8 

Tradeoff  &  Optimization  8-4 

Tradeoff  and  Optimization  5-4 
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